Dot notation for json-like objects in python

This post gives a class Dotable to make nested python dictionaries (json-like objects) accessable using dot notation. I then use this to fill in strings which use ruby’s #{} format.

Why it’s not there by default

I could’t find a PEP for this, but it’s easy to see the logic to it, for example consider the unruly behaviour of

d = {'get': 'me'}
d['get'] == d.get # ??

Anyway, practicality beats purity, so here’s my solution:

class Dotable(dict):

    __getattr__= dict.__getitem__

    def __init__(self, d):
        self.update(**dict((k, self.parse(v))
                           for k, v in d.iteritems()))  # in py3 use .items

    def parse(cls, v):
        if isinstance(v, dict):
            return cls(v)
        elif isinstance(v, list):
            return [cls.parse(i) for i in v]
            return v
  • __getattr__ is the important part which means we can use to dot notation 1.

  • The parse method is quite general, it takes any json-like object and returns an object whose dictionary part can be accessed with dot notation. I thought this was quite a neat solution (not just because it’s recursive).

Note: Some people on StackOverflow have posted about using dot notation before (2,3,4), but I wasn’t happy with these implementations (to be honest I didn’t look too deeply, and think this is a simpler solution). If you have a nicer way, please leave a comment and let me know!

Example use:

In [1]: d1 = Dotable.parse({'a': [{'b': 3, 'c': 5}]})

In [2]: d1.a
Out[2]: [{'b': 3, 'c': 5}]

In [3]: d1.a[0].b
Out[3]: 3

In [4]: d2 = Dotable.parse([42, {'a': 'shrubbery'}])

In [5]: d3 = Dotable.parse(7)

Why I wanted it:

I wanted to populate strings which contained strings with ruby syntax #{} from text files, where blah a json-like object.

Before you start writing something with ~~evil~~ eval and regular expression (then you’ll have two problems!) to fill in #{}

You can use format (unless your using python 2.5):

In [11]: '{d1.a[0].b}'.format(**locals())
Out[11]: '3'

I lied about not using regular expressions, you need a little one to remove the #s:

In [12]: s = '#{d1.a[0].b} #{d2[1].a}'

In [13]: import re

In [14]: re.sub('#({[^}]*})', r'\1', s)
Out[14]: '{d1.a[0].b} {d2[1].a}'

In [15]: re.sub('#({[^}]*})', r'\1', s).format(**locals())
Out[15]: '3 shrubbery'

Hope this helps.





Tagged with:
Posted in Uncategorized

Another git(hub) workflow

This is another git(hub) workflow1 which I find works really well, along with all the commands you’ll need to make it happen. It only has three simple rules, but stick to them and you’ll find managing git will soon become a pleasure not a chore.

Rules of the game:
  1. Populate upstream with pull requests
  2. Update local master from upstream
  3. Update branches with rebase
Branch management:
Configuring git:


Populate upstream with pull requests

Never push directly to upstream

Make use of github’s pull request (to upstream):

  • alerts everyone in the team that a merge is being requested
  • a good oportunity for a peer review, and to keep everyone up to date with what you are doing
  • it can be easily tested by everyone in the team (see manaing remotes for a trick here)

Once merged:

  • everyone can see in the topology of the network that something been merged, who did it and who had a checked it.
  • everyone can see your comments and discussion on the branch, so it’s a good idea to be verbose.

Note: many teams do push small commits directly upstream, and this does lead to a slightly cleaner network. However, I think it is good practice, and easier to set a blanket policy (which in practice this means: only do so if someone has borked something and you know what you’re doing, see troubleshooting).

Update local master from upstream:

Your local master is a copy of upstream

Everyone has a backup of upstream from the last time they grabbed it

git checkout master  # got to the master branch
git fetch upstream  # get the latest
git merge upstream/master --ff-only  # it's important you're in your master branch

Note: the --ff-only (fast forward only) ensures the commits are added in the same order as upstream. This should be superfluous: so if it complains then you’ve done something wrong.

Usually you would push straight to origin (your gihub fork):

git push origin

This means everyone can see the last time you fetched from upstream.

Branch management

Create a new branch

For each topic of work, create a new branch (off of the the latest master).

git checkout master
git checkout -b describe-work-doing
# do some work
git commit -a -m 'describe what is in this commit'
git commit ...
git push origin describe-work-doing  # backup to github (for pull request)
Unrelated features/bugfixes in different branches

Note: It’s a subjective call precisely what a “topic” is. For example, add_feature_a would be a good choice, but my_days_work would be terrible.

It might be you make an immediate pull request, or it could be that this particular job takes a while (see updating your branches), or you want to tidy up the commits (see squashing commits).

Making a pull request

Visit your branch on the github page and make a pull request. 2

  • Add a concise desciption of the changes which you have made (link to any bugs which you have fixed)
  • Mention to developers if they need to do anything once to see this change re-compile, db:migrate etc
  • At mention developers whose opinion you would like

Updating your branches:

Once you’ve updated your local master (from upstream), you should update your other branches to see these changes, and ensure your code still works:

git checkout branch-name
git rebase master

Note: rebase rewinds your commits, applies the commits/updates from the latest master then reapplies your commits.
Some people favour merging here, but for one thing you get a cleaner (easier to reason about) network when rebasing.

Update branches regularly

This means conflicts (if there are any) are resolved as they appear, rather than in one massive hit much later. If you’re having trouble you can ping the other dev while the change is fresh in their mind.

There may be merge-conflicts at this stage (if so, then you wouldn’t have been able to merge in with the upstream codebase anyway, so it’s usually best resolve them locally).

When pushing to github you’ll need to do a force update.

Resolving merge conflicts

This isn’t usually too bad, if unsure cancel the rebase, see instructions in terminal…

Back up (optional)

If it’s a particularly large update/merge conflict, it’s often useful to create a backup branch with your changes as they are currently (before the conflict):

git rebase --abort  # give up on the rebase
git branch feature_x_backup  # make a backup branch

Attempt the rebase, and test that it’s still working then you may delete the branch.


Don’t use “yours”/”theirs”. You’ll lose code. Very often you’ll have both made edits to the same file, and will want to keep them both.



Resolve each file, and once resolved, mark this file as resolved (via git add file_name).

Tidying a branch (squashing commits)

First check how many commits you want to squish (you’ll have a chance to review later, but check with git log). Suppose you want to squash the last 3 commits 3:

git rebase -i HEAD~3

this will open editor, change all but first pick to squash, and change the commit message. I promise this will make more sense once you’ve done it a few times…

If you’re squashing commits on a branch already on github, just like with normal rebasing, you’ll need to force update.

Force pushing to github

Here be dragons, use caution:

  • If there are others using you branch (i.e. doing work off it), consider not doing this (THEN WHAT?)
  • If you have a pull request open, ensure it won’t be merged by another developer (close it or simply comment a warning saying you are rebasing)

Since you are rewriting history you need to force the push:

git push -f origin feature_x

Note: a github pull request will be updated with the new commits (and the old ones will be deleted).


Adding commits to a pull request

Usually you want to avoid this, but sometimes collobations is sometimes a necessary evil (good thing?).

It’s usually best to tell the developer that you’re going to do something from their pull request (see force push). At the very least, comment on the pull request to let them know.

If you’ve set up pull requests, then checkout theirs (where # is the number from github) and make yourself a new branch from it:

git checkout upstream/pr/#
git checkout -b descriptive_branch_name
# do some work

You can now treat this as a regular branch (remember to close the old pull request). If the old pull request is merged it will cause problems.



Whoops: I’ve commited to master

Just make a branch from here, and continue working:

git checkout -b branch_name

To fix your master you’ll have to delete the last (few?) commits:

git checkout master
git log  # see how many commits since the last pull request
git reset --HARD HEAD~3  # delete last 3 commits

and then update from upstream.

Note: Provided you do this in master, and have backed up your branch deleting the last few commits shouldn’t be too dangerous. Nevertheless, breath holding is mandatory.

Whoops: I’ve merged into my local master

We essentailly do the same as above, that is, delete the last few commits and update from upstream.

Configuring git:

Initial set up

If your the first person in the team to set up:

  • Create a git repository for your work
  • Create a (private or public) project github repository4, this should use an email different from any of the developers.
  • Set up remote upstream, and (just this time) push directly to this (git push upstream).

Everyone else

  • fork from the main project repository5.
  • clone (from this fork!)
  • Set up remotes (e.g. upstream)

Managing remotes

Add collaborators repos

We should have upstream as team/project, I like to inspect Bob’s repo at bob/project (look up the url from their github page):

remote add bob

Just like we do for upstream, I can have a look at what he’s been doing (if he needs some advice):

git checkout bob/feature_y
Fetch all pull requests

For more easy checking of pull requests before merging, you can append:

[remote "upstream"]
    fetch = +refs/pull/*/head:refs/remotes/upstream/pr/*

to your .gitconfig file. This means that when you fetch from upstream, it downloads the branches of every pull request as upstream/pr/# (where # is the pull request number from github).

git checkout upstream/pr/#
# test things

Abbreviating git commands (aliases)

I like to save some typing and use the following aliases (add these to your .gitconfig file):

    co = checkout
    com = commit
    br = branch
    st = status
    re = rebase

Which enables you to write commands more concisely.


Add some colour, by adding this to your gitconfig:

    ui = true
Tagged with:
Posted in Uncategorized