The overhaul is nearly complete! The Financial Firsts site is currently in testing, and I’m patching as needed, but the core seems to be coming along nicely.

The ‘Member’ Keyword

Introduction

Mogu’s current functionality is great for more static sites. Creating an application with nice effects, even adding users, is easy to do. Sadly, the user/group functionality is still absent. It has taken a lot of brainstorming to figure out how to implement this while keeping the syntax for MoguScript tight.

Interacting with multiple data fields at the same time is an obvious requirement. Pretty much any “group”-centric applications has moderators that need to be able to, well, moderate. Doing that requires some functionality to view and edit their consituents’ data.

Introducing the “Member” Keyword

To solve this problem, we’re implementing one keyword to rule them all: MEMBER.

Depending on its context, this keyword will mean multiple things in regards to how it functions, but should only mean one thing to you. Creating a dynamic widget is a three-step process.

First, you have to set permissions in a piece of data’s storage policy.

policy personal_info
    type        :   hash
    read        :   user moderator
    write       :   user
    encrypted   :   yes
end policy

The above example would allow a user and their moderator to view fields in their personal_info hash, but only allow users to edit this information.

For our example, let’s pretend we want to create a table allowing moderators to view the height of all their group’s users.

We then need to create a container for this information

Let’s create a simple table of “username: height” lines.

widget group_heights
    type        :   container
    access      :   moderator 
    content     :   member personal_info
    template    :   height_row
end widget

Above, you’ll see we did something that seems to conflict: created a container widget and assigned it content, which would seemingly have no effect. The trick here is that the content will be applied to the template, and the template will have nothing to do with the widget ‘group_heights’. You’ll also notice that even though ‘personal_info’ is a hash field, we did not assign it any arguments. That’s fine. The identifier alone is enough to look up the access rights; the actual assignments come next:

Next, create the view for the dynamic information

template height_row
    type        :   container
    children
        :username
        :height
    end children
end template

template height_row:username
    type        :   text
    content     :   member personal_info username
end template

template height_row:height
    type        :   text
    content     :   member personal_info height
end template

Above, you can see that just as if we were creating a similar widget set with static data, we’ve done so with templates. It should be pretty clear how this works from a high level perspective.

Behind the Scenes

How will Mogu resolve this information?

When Mogu sees a template and content applied to a container, it first checks the access rights of ‘content’ to make sure that the user instantiating the widget has access to this content.

Then, it looks up the list of all members of the group. For each of those members, it will apply the specific data to anywhere it is needed within the template set.

In theory, this could be expanded to have even more uses, such as showing all elements in a list field for every user (for instance, showing a user a list of all of their interactions within an application), and can also work for editing user data by applying actions. If above, the moderator had write access, we could have also done something like this:

template height_row:edit
    type    :   input
    content :   member personal_info height

    events
        when keyup {
            set member personal_info height to own text
        }
    end events
end template

Security Concerns

Ownership of data is, of course, huge. Currently, this schematic offers users very little control of what can be accessed by moderators, and that’s bad. With a little event magic, however, this can be dealt with. If you want to give users the option to hide this data from moderators, for instance, you could create an option that works as follows:

policy private_personal_info
    type    :   hash
    read    :   user
    write   :   user
    encrypted:  yes
end policy

widget make_private
    type    :   text
    access  :   user
    css     :   "linkCursor" # make this appear as a link when hovered

    events
        when clicked {
            set user private_personal_info username to user personal_info username
            set personal_info username to "HIDDEN"
            set user private_personal_info to user personal_info height
            set personal_info height to "HIDDEN"
            set user personal_info_store to "private_personal_info"
        }
    end events
end widget

To display this data to the user, you would then do something like this:

widget my_height
    type    :   text
    access  :   user
    content :   user user personal_info_store height
end widget

In the above example, “user personal_info_store” would be resolved first, followed then by “user private_personal_info”, which would actually retrieve the height. The group moderator would thus only see ‘HIDDEN’, but the user would be able to access their own data.

Fun!

end blogpost

We’re very close to the Borel release of Mogu now. So close, in fact, that I’m doing that thing where I wake up at 5:00 in the morning excited about Feature X.

New Syntax

Forget everything you knew about Mogu syntax before. I’m hoping you didn’t know anything at all.

Because we now have a fully-realised lexer for Mogu, we have a lot more flexibility. In short, though, you will no longer need to know Python to write Mogu. Mogu is a language all on its own.

Here’s an example Hello, World widget:

widget hello
    type    :   text
    text    :   "Hello, World!"
end widget

You can now declare children and events in the same block, also, using a much more natural language.

For example, let’s pretend that Widget A is a red button, and widget B is a blue button. When you click ‘widget a’, you want widget b’s text to change to ‘Red’, just to confuse your users. Let’s also pretend that both widgets are located in a div called “confusingbuttons”

widget confusingbuttons
    type    :   container
    children
        button_A
        button_B
        reset
    end children
end widget

widget button_A
    type    :   text
    css     :   "linkCursor redBG"
    text    :   "Red"

    events
        when clicked {
            set widget button_b text to "Red"
            set own text to "Red"
        }
    end events
end widget

widget button_B
    type    :   text
    css     :   "linkCursor blueBG"
    text    :   "Blue"

    events
        when clicked {
            set widget button_a text to "Blue"
            set own text to "Blue"
        }
    end events

widget reset
    type    :   text
    css     :   "linkCursor"
    text    :   "Reset Buttons"

    events
        when clicked {
            set widget button_a text to "Red"
            set widget button_b text to "Blue"
        }
    end events
end widget

Math!

Mogu can now do math using the above syntax. The specifics will be covered in the docs once the release drops.

Infinite Command Recusion

Mogu commands are now resolved in reverse, allowing you to create ridiculous chains of dynamic commands.

set widget foo text to data bar widget baz text

is perfectly valid (if confusing and unmaintainable).

This is the equivalent of doing something like:

set slot A to widget baz text
set slot B to data bar slot A
set widget foo text to slot B

This gives you more control over your application data.

Python API

Since Mogu no longer requires Python knowledge in order to write MoguScripts, it now has its own API to make script generation easier. The details on this will be given when the release hits, but in short:

from moguio.ScriptWriter import WidgetWriter

hello_widget = WidgetWriter(
    "hello_world", 
    {"type" : "text","content":"Hello, World!"})
print(hello_widget)

The above would generate the required Mogu Script. This supports events and children, both at instantiation and late binding occurences.

MoguString

Using MultiString, the Python API has an easy way to translate back and forth between standard script syntax, and the integral syntax used during the lexing and database import process.

from moguio.MoguString import MoguString

mogustr = MoguString("script", "widget hello_widget")
mogustr.translate("integral")

This allows you to easily roll your on API bindings for Mogu.

Better Builds

Mogu is easier to build than it used to be, and the CLI and API take full advantage of PyPi in order to install these components in a more standardized manner. The output from the build process has been cleaned up to be more informative, and allows for more customization of the install.

Additionally, Mogu’s integral syntax is built at compile-time, making it specific to your system. The only bad news here is that you should never, EVER refer to any of Mogu’s integral syntax by its integer. You should always use the enumerated types (if you’re hacking the C++), or the

syntax.as_integer(...)

method (if you’re hacking the Python).

This also allows us to keep the syntax updated without having to change any references to any of the syntax anywhere else in Mogu’s code base.

Better Import Structure

Mogu is now a lot smarter when you import .mogu files. Before, especially with the horrible syntax, it was easy to make a mistake that would break your Mogu app. Now, Mogu tries to catch errors in your application scripts before that happens.

During the import process, Mogu tracks references to all of your different object types (widgets, templates, validators, etc.). Once all of the files in the given path are lexed, Mogu will look for any symbols that have undefined references.

If so, it will tell you where all of these references take place, and fail to import.

Likewise, if you define symbols that are never used, you will get a warning, but can still import your project.

More Efficient Storage

Mogu is now more efficient when storing your application data in Redis. Instead of storing standard syntax as strings within the Redis instances, all of this is converted to integral syntax first. This makes for shorter ‘strings’ in Redis. This also makes it easier for the engine to parse your commands, as every static mogu keyword is represented in its integral form already, saving from one more conversion.

More Efficient Redis Interactions

The engine now uses pipelines instead of atomic queries, which speeds up the communications between Mogu and Redis.

Multiple Redis Instances

Mogu can now split your application up into different Redis instances (or different databases within the same instance). This makes Mogu projects more scalable, and can allow you to take advantage of a Redis cloud, as well as keeping your user’s important, secured data separate from your static, unimportant style data.

More Sane User Database

We’ve scrapped the idea of the session storage schematic from before. It was difficult to maintain, slow to process, and error prone.

Be on the lookout for more details as the release approaches. We’re very excited about these milestones. It’s been quite an arduous few months!

The new importer has been written.

Changes to the Command Syntax:

• No longer have to differentiate between files and directories.
• Command now accepts a mixture of files and directories
• Filenames must end with Mogu

“` mogu import [path [path…]]

mogu import a_file.mogu

mogu import a_directory

mogu import a_file.mogu a_directory /path/to/other_directory some_other_file.mogu

“`

Import Sequence:

When importing a directory, the following happens, in this order:

1. Subdirectories imported recursively
2. Each file in the current directory is lexed, parsed, and symbols are registered into the global symbol table.*
3. The symbol tables are searched for symbols that are referenced but never defined, or defined but never referenced.
4. Any undefined symbols cause importing to halt. There is no way around this. Mogu will no longer allow you to import an application that is obviously broken.
5. Any unreferenced symbols generate warnings — you may continue manually, or with the “—assume-yes” flag. While harmless, unreferenced symbols will clog up your database and namespace.
6. If all is well, the results will be converted into RedisObject types, and finally:
7. The RedisObjects will be written to your redis instance.

Except Sessions

Because sessions are not hand-written, they can be more compactly imported and exported, and a separate handler is going to take place for those. That said, there is currently no way to import sessions.

Don’t Export, Either

The exporter has not yet been rewritten. Therefore, don’t export anything, because if you do, you’ll get old Mogu which will be completely unusable.

But I’m Not That Mean

Once the new exporter is written, it will contain contingencies to write old databases in the new format. Hooray!

In short, the changes are fantastic, and you should be excited, but you shouldn’t get too excited until they’re finished.

Long time, no update. Why? Sadly, because there have been about seventeen bajillion other projects happening with deadlines sooner than those affiliated with Mogu.

We’ve acquired more resources to aid with the development of Mogu, and are now in the process of continuing for the Borel push.

Borel is going to be a complete departure from the original Mogu, as we’ll be abandoning the ridiculous syntax and turning MoguScript into a more traditional language. I’m working on the lexing of this script, and our new teammate is working on upgrading the engine to match.

New syntax looks like this:

widget my_widget
    type : text
    content: "Hello, World!"
    css : "some css classes"
    events
        when clicked {
            set own css to "some other classes"
            increment data num_clicks
        }
    end events
    children
        child_a
        child_b
    end children
end widget

That’s right: No more mucking about with syntax wrappers or python dictionaries, or having to separate widgets from their events and their children. No more “listener”, “message”, bull in event creation.

In addition, upon importing, Mogu will translate everything it can into inetegral form to speed up the runtime substantially.

The main lexer and importer is done, and we should have Borel up to snuff within a couple of weeks for building. Huzzah!

You can check out the Lex branch if you’re interested in seeing the changes.

Cheat @ Joe's: preJS - New Repo

cheatatjoes:

Nothing to do with Mogu. I’ve had to take a small break from Mogu to work on another project for a few weeks, which is a JavaScript application.

I don’t really like using JavaScript, and having been immersed in C++ for the past ten months, and doing project management in Markdown, I really wanted…

Download it here.

To upgrade, unzip the file and do: cd Mogu && sudo make upgrade

WARNING: NOT BACKWARDS COMPATIBLE!

Here’s the Change Log, followed by explanations on how you can fix any current projects to make them work with the new syntax mapping method.

• Removed enums.h, parsers.h, and all dependencies
• Unify all enumerated syntax into one enum
• syntax map is now extern, making it much more efficient
• Converted all syntax uses to new methodology
• Add “check-syntax” CLI command
• make install-cli, make, and make syntax now create syntax maps on build

The New Syntax

In order to make things (way) more efficient and easier to maintain, I’ve unified all of the enumerated types used as syntax. They are all now part of the same enum, and the lookup (and reverse lookup) is now a global value that is only created once per instance of Mogu running on your server (not once per user!). Here are the benefits of doing this:

• No longer need to instantiate an enum parser every time one specific value needs to be looked up
• No longer need to edit more than one file when adding (or removing) syntax
• Easier for forked repositories to create their own Mogu dialects for use in their projects.
• Easier to find typos in Mogu applications

Why Not Backwards Compatible?

Before, the word “index” could mean many different things depending on which enum it belonged to. Now, all of those enums have been consolidated. This meant there had to be some changes in order to make this simpler to work with and ensure that syntactical meanings were explicit. In order to ensure rapid lookups for both string-to-enum and enum-to-string, I had to ensure these were 1:1 as well, which means I can no longer be forgiving when it comes to interpreting common human errors.

This also means when writing widgets, you must make many fewer mistakes. It also means that Mogu can be built to check for these mistakes. And work on that has started:

The New Command

mogu check-syntax will now scour your entire database for enumerated values and widget states that don’t exist in the current syntax. It will output a report called syntax_check.txt, which details the bad values and in which widget they were found, allowing you to change them.

If you’ve already built a Mogu application and have a lot of widgets to edit, I recommend the use of the sed program. Here’s an example script I wrote to fix my own database after exporting it:

for f in 1220/widgets/*.mogu 
do
    echo "Fixing $f"
    sed -i "s/{increment_index}/{increment}/" $f
    sed -i "s/{test_text}/{test}/" $f
    sed -i "s/{stacked}/{stack}/" $f
    sed -i "s/{match}/{test}/" $f
    sed -i "s/{decrement_index}/{decrement}/" $f
    sed -i "s/\\\$current_index\\\$/\\\$index\\\$/" $f
done

If you (right now!) export your application database using mogu export 1220, you can run that script as-is and it will probably fix any potential problems for you, unless you’ve made some typos. In which case, you’ll now know what they are!

Other Fun Facts

This upgrade resulted in the next loss of almost 1000 lines of code, and approximately 300kb in the executable binary (if compiling at the highest optimization level).

I am still running tests to determine which optimization level runs the fastest, but honestly, they all seem to be about the same. It takes me approximately 4 seconds to instantiate over 1000 widgets, with varying degrees of complexity. I haven’t done any more standardized benchmarking. Let me know if you have, and what your findings are!

I have not done any memory checking on this yet with valgrind. There could be some memory leaks, but I also have not experienced any issues with our production application, either.

Adding Syntax

If you want to add syntax, just edit the syntax/syntax.pre file. The left-hand assignment is the enumerated name, and the right-hand assignment is the string equivalent.

As an example, when I start adding mathematical operations, I’ll be doing it like this:

plus:+
minus:-
mult:*
div:/

(or thereabouts). Then, just run make syntax, and ka-blamo. You can now use your new syntax.

The Syntax Map

Because I use this blog to talk about the development itself, and not just the changes, I’d like to take a second to explain how the syntax map works.

I created a SyntaxMap template class, which uses two unordered maps, one for converting enums->strings, and the other strings->enums. Unordered maps are faster for lookups than ordered ones (and available in c++11).

The map has an overloaded getmethod, which takes either an int, enum, or a string, and returns the corresponding value (enums and ints return strings, and vice-versa).

I then created the SyntaxReader class, which processed the syntax/syntax.pre file, and the SyntaxWriter class, which writes three different files from that information.

The generated files are syntax.h, which declares the enum and exposes the global syntax map to Mogu, syntax.txt which is the instantiation of the syntax map (and included in the main.cpp file at compile-time), and syntax.py, which creates a python list of the valid syntax strings.

When running make syntax, an executable is created in the syntax directory, which reads the .pre file using a SyntaxReader, writes the three files produced with the SyntaxWriter, and then copies the generated files to their required location.

It was a fun project and ultimately not too difficult. I’m still trying to optimize the widget generation, and this was a big step in that.

Summary

The unification of enumerated syntax will allow Mogu to become more powerful, and much easier to work with in the long run. I’ll be able to start phasing out the wrapper syntax and opt instead for something a little less awkward, and also will be able to add new features much more quickly and reliably.

I do apologize if it breaks one of your current applications. However, I have 1700 widgets in my application currently, and it only took a few minutes to fix with the script above. Please let me know if you have any questions.

I’ve added the following two entires to the wiki, which were desperately needed:

Interpreter Automaton describes, for those interested, exactly how Mogu parses input.

Syntax describes how to use Mogu syntax in depth, and why it is the way it is, and why it makes your life better.

The second release candidate for Mogu Borel has been tagged and is ready for testing. It is not yet feature complete, but it’s nigh on close, at long last!

• Remove Nodes namespace
• Remove “policy” as a widget attribute
• Add “properties” as a widget attribute
• “input” widget content is now empty text, instead of standard text
• Mogu now uses an internal userid instead of the user’s encrypted login.
• Now compiles with the -std=c++0x flag, for C++11 features.
• “make purge” now removes the generated configuration file.
• Add “memberlist” widget type and its necessary handlers — untested
• Add “change_group” action, which changes the active user group
• Fixed a problem with upgrading to the newest version of Redis
• cityhash 1.0.3 now included in the repository for backward compatibility. If y
• “mogu find-orphans” works again
• “set_index” also emits “hidden_changed”
• Add “test” action, which works much better than either of its predecessors, wh
• Reworked all widgets to separate them out into sensible classes

I’m a little confused as to some of the boolean logic I have set up for the “match” and “test_text” actions. They’re very strange and not very useful.

I’m currently working on a generic “test” action whose listener can be just about anything, and whose message can be just about anything. The test will be to see whether once parsed at runtime, those two values are equivalent. If so, the ‘succeed’ signal is broadcast, otherwise the ‘fail’ signal.

Why i didn’t set that up to begin with is a mystery of epic proportions. A new release candidate is really en route, I swear. A lot of the last few features are conceptually finished, but have not been fully tested in an actual environment yet. Patience!

-tom