One solution is to produce complex regexes that return a different set of variables for a url pattern (i.e. you want to be able to query a url by both the entity's id and its english name: /foo/{foo_id}/$ Or /foo/{foo_name}/$). Though this use case may not be universal, I find it annoying to have to create a complex, error-prone regex or define a simplistic one for each needed url.
Another downfall of the Django urls.py is that it requires you to spell out a new url for every single view you have of your code. Several plugins are available to attempt to solve this problem for you, but I have not liked any of them. The reason I don't really like them is 1) requires an external dependency for something that should be really simple, and 2) usually takes control of the internal routing design that Django ships with. Neither of these are really that bad since Django is very customizable and in a sense is made for this sort of behavior; however, I do not like it. Therefore, I have created a means whereby to get the url functionality I wanted without having to download a third party app (and it is contained all within about 200 lines of code, including comments and documentation).
This tutorial supposes that you are familiar with Django routing. If you aren't, it might help to read the Django tutorial on routing first.
You can see the source code for this also at my github page.
Requirements For Custom Router
Before I walk through how to create your own custom router, I want to list a few requirements I had for this module:
- Foremost it had to be self-contained, meaning that it couldn't rely upon another python package to work.
- Any routing had to be done through the default Django route system as it has already proven to be sufficiently performant.
- Error checking of urls was a must (why Django doesn't come with some sort of error checking beyond the regex compiling module is beyond me). Error checking in this case means that the base path of the url (a path without any regex patterns included) was different for each new urls. Thus you wouldn't ever have the infuriating problem of wondering why you aren't seeing what you expect on very similar but slightly different urls.
- Had to be able to keep the same functionality of original url() function that ships with Django.
- Must be able to create custom routes as well as automatic routes for the application. That means that it will produce <app>/<view>/* urls were the star will be treated as variables split by the /, and any other custom url you wish.
- View routing must be attached to the view. I hate having to go back to the urls.py to figure out what the url is I am looking at to see the grouping I had used for the variable I need.
- Must be easy to use and require very little configuration.
Most of the requirements above were met with my first attempt within an hour's worth of work. It is really quite simple, and I was able to make 149 different unique urls with the following urls.py definition:
urlpatterns = patterns('',Looks much cleaner than the normal 150 lines of excessive regex that would be required. The next section will detail how to write the router used, with the full code pasted at the bottom.
url(r'^admin/', include(admin.site.urls)),
url(r'^accounts/', include(accounts)),
url(r'', include(routes.urls)),
)
Using Django for Inspiration
The basic requirements of making this router self-contained and not circumvent the normal Django routing requires that all the routing creation occurs before Django ever serves up a page. Several things happen when Django is first loaded, and the event we are interested in most is how the settings.py is set up to be global. If you are not familiar with the Singleton pattern, just know it is a way to make one - and only one - instance of an object. Python makes this very easy because modules are, in Python, objects themselves. This allows for very simple singletons, and it is a pattern that Django uses to make the settings Global and singular (see the Django github for the settings moudule for more details).
Because Django loads things once and only once at the initialization of the system, it does not have to worry about linking to sources and hoping that they have not been moved, while still having the advantage of storing things in an encapsulated object and not inside the global namespace. This provides the benefits of OO programming with the accessibility of functional programming. Elegant and simple. We will do the same for our router.
So the question arises: How do we get Django to scan the views for the routes that we will be adding? At this point you may give up and think that this isn't really worth it (I certainly did), but the reality is that this problem is not a hard problem to solve. To get Django to scan, all you need to do is import the modules that hold the views into the urls.py module. That is it! Python will scan those modules for you. Using this fact will make almost all of our requirements possible. So, first thing we need to do is import the modules that hold our views into urls.py.
Importing the modules we need solves the most basic problem of getting the routes into a position to be added to urls.py. However, this does not establish an easy way of creating app/module/* routes since simply importing the files would require that we do lots of writing of code to add them and removes the automatic nature of adding the routes to urls.py. We could simply write the app/module/* in the view that will be calling it, but that just seems against the auto-magicness of python. We can do better, and we will.
Setting Up a View Based Routing Paradigm
Since one of our goals is to make routing definitions found on the view that it is routed to, we will take advantage of the fact that class-based views are the new way of making views in Django. Don't worry, the old functional views paradigm will also be supported, but it will be clunkier or less flexible. To make this work, we will simply define a new variable on the view:
routes = ...
What should routes be set to? I went through a number of ideas, including an overly complicated tuple of tuples, which in the end was abandoned for a pythonic (and much easier) way of doing things: dictionaries! Because we decide to use dictionaries right off the bat, we are free to change the structure as we go along and no longer are weighed down by needing to keep track of what order our route definition is defined. If you ever think of tring to define configurations using a static list, just step back, slap yourself awake, and realize that dictionaries are the best thing you could possibly use in this situation.
That being said, I decided upon a routing definition structure as follows:
routes = {"pattern" : '',
"map" : [(.)],
"kwargs" : {}
}
To explain each in turn, remember that we are making it so that each route can define variables in the url to be of one type or another (ie. id based or name based for lookups). Therefore, we need to allow for a way to define the pattern once and then add new regex groups to that pattern. This allows us to also error-check the pattern to make sure that we are not registering the same url pattern more than once unwittingly.
Pattern
The pattern is simply a string that uses the syntax for the format() function of a string to change the curly braces (i,e, {}) inside that string to the positional value of the args passed in to format(). For example, if we want to make a route of foo/{foo_id}/ and foo/{foo_bar}/ we make a pattern:
foo/{}/
Simple!
Map
Because we are defining string templates with pattern, we are going to create a list of lists that can be used to substitute those curly braces with the desired regex pattern. These lists must have the same number of regex patterns as the number of {} in the string in order for this to work, or it will throw an error. The map variable for our pattern above should therefore look like the following:
[('foo_id',), ('foo_name')]
This produces to distinct url patterns:
foo/foo_id/
and
foo/foo_name/
Note that these are not regex patterns, but simple strings. This is intended to keep it simple, but you can add a regex pattern in there to capture a variable just fine.
Kwargs
The typical name for passing around key-value arguments is to name the dictionary that Python creates as kwargs. To keep with this tradition, we will name our configuration dictionary the same. Any arguments that you need to pass on to the underlying view should be given in the kwargs. This directly correlates to the kwargs argument of the url dispatcher in Django.
Miscellaneous Notes
Because we are using a dictionary to setup our configuration, we now can make use of the 'key in dict' pattern where we can search if a configuration has been defined. By doing so we can omit and add configurations for a route as needed.
One such example is the name variable defined in a url dispatch object. The name given to a url dispatch object is used in reverse url lookups within templates and can come in handy in many ways. Because name is such a common variable to use in so many applications, it has a high chance of clashing with predefined variables in the kwargs section our our route definition. To circumvent naming collisions (and to make the intent of adding a name for reverse lookup to a url more clear), I later added the 'django_url_name' route configuration option to my router. I was able to add this in to my code without it affecting any other aspect of the routes setup. The same can be said about any new routing configuration that you may want to add in the future.
Creating the Routes Table
Now to the good stuff. The routes table will make use of the singleton pattern described above. This was heavily inspired by Django, including the workaround of LazyRoutes (which will be discussed later). At the bottom of our routes.py, add the line:
routes = Routes()
Routes() here is referencing a class that we have not yet defined. Go ahead and define it anywhere above that line:
class Routes(object):
Now that we have defined the singleton (routes = Routes()), and the class, we are ready to start adding the structure of our router.
I will not go into full detail about all the different ways you can set up your Routes() object, but I will cover three aspects of it: the initialization (which will be where we automatically add the app/module/* routes as well as any class-based view routes), error checking of routes, and the add() function.
The first thing to do is make sure that the routes are unique. To do this, we will add a set object to the Routes module definition (not the routes instances). This is significant since we will be using this to keep track of all routes added, either by Routes or LazyRoutes (again, discussed later). To do so, you will write a class that looks like this:
class Routes(object):
tracked = set()
routes = []
You will also want to add the routes list at this point since want to make a single source for both Routes and LazyRoutes to place their urls. Now, when you add a pattern to the routes table, it can check the tracked set to see if it has already been defined by calling
pattern in tracked
If pattern is in tracked already, then this gives us a chance to throw a meaningful error, one that can be used to denote the duplicate pattern. I have wrapped all this in a function contained within the Routes class as follows:
def _check_if_format_exists(self, route):
'''
Checks if the unformatted route already exists.
@route the unformatted route being added.
'''
if route in self.tracked:
raise ValueError("Cannot have duplicates of unformatted routes: {} already exists.".format(route))
else:
self.tracked.add(route)
Now that the routes base pattern is unique, we can with confidence add route patterns to each view and know that we won't inadvertently step on a route we already defined.
With the above function we now have an adequate check to use in our add function. We create our add function in such a way that we can pass the pattern, the map, the function to call, and the kwargs to add to the url. Note that I said the calling function. Here we define a way to add function based views and their routes. My add function looks like the following:
def add_url(pattern, pmap, ending=False, opts={}):Since Django's url dispatcher literally stores the function signature to use when routing a url to a view, we can safely add any function that fits the url dispatch function parameters. If you look at the source code for Django View object, you will see that the as_view() function that is required to be passed in to a url dispatcher object literally returns another function called view. This function fits the old function-based view pattern of:
url_route = '^{}{}'.format(pattern.format(*pmap), '/$' if ending else '')
if "django_url_name" in opts:
url_obj = url(url_route, func, kwargs, name=kwargs['django_url_name'])
else:
url_obj = url(url_route, func, kwargs)
self.routes.append(url_obj)
def view(request, *args, **kwargs)
Since the class-based views are just passing this function as the view, it is therefore clear to see that any function with this pattern can be passed safely. Note that in order for it to work it must return a django.http.HttpResponse object, but you could register a function with this pattern and almost get away without it. So, because we know that all we really need is a function, we now have the ability to add any view function to our routes. Isn't that great!?! An example of what I mean is as follows:
Here is your view function:
def showMeAll(request, *args, **kwargs):
....
All you need to do to add it is to first import routes, and then add it as follows:
routes.add_url('foo/{}', [('foo_id',),('foo_name',)], True, {})
The ending variable of the add_url() definition is to add the '/$' at the end of the url, thus eliminating mistakes that arise from not including the pattern end clause and preventing the need to make sure all urls end with a /. The opts is simply the kwargs as described in the class-based view route table.
The LazyRoutes Object
The LazyRoutes object pertains to the automatic loading of view-based classes into the routes table along with the automatic addition of app/module/* routes. The detail behind how to make an automatic loader is also heavily influenced by how Django registers apps and Models. The way Django loads apps is confusing, and it took a little bit of trial and error to finally figure it out, but if you want to see more about it, here is the link.
From what I can make of it, since Django imports everything it needs at once before it ever serves up a page, there are times when recursive import statements can become a problem. What I mean by that is say that, as in our situation, we need to load the views modules when we have custom routes to add to the urls.py, but we also want to automatically add the app/module/* routes along with these custom routes. To automatically create these routes, we need to create them through introspection upon the creation of the Routes object. Simple enough, so what is the problem?
The problem arises by the fact that, when scanning the modules for views, we may come across a line of code like the following:
routes.add_url(...)
This is for a functional route, one that cannot be added through introspection upon creation. What are we to do? We do what Django does and create a LazyRoutes object. Technically this is not a real Lazy Object because it creates the url objects when they are found, but the idea is that they are not added to the Routes object that is still being created. The LazyRoutes object is used to store url patterns in the Routes.routes list (remember that the class definition Routes is its own object and is not an instance of Routes). This LazyRoutes object seems to act as though it adds the routes definitions defined by the routes.add_url() function after the Routes object has been created and has finished making the app/module/* routes. In reality, it was adding them to the master list all along. But this is a detail that is needed to be known in very few, if any, circumstances.
The added bonus of having a LazyRoutes object is that we can also completely circumvent the automatic Routes behavior if ever we wished and just stuck with LazyRoutes and only making defined routes accessible. Whatever your style of coding is will determine whether you will use it in this way.
How to Introspectively Create App/Module/* Routes
Finally, we will discuss the trickiest part of this whole routing setup. So far it has been very easy, no? Hopefully you will have figured out that to add a class-based view would mean iterating over the routes dictionary you defined in the view class, but if not, here is a hint that that is what you should do. This aspect makes it so that you don't really have to even do that, for as you will see, you can add a function called add_view to your Routes definition that will take advantage of the introspective magic we are about to cover to add these views automatically (meaning you don't have to register them with routes.add_url()).
Python comes with a bevy of really cool tools for introspection (something made very easy since it is an interpreted language). Since Django already requires us to list the importable app names of our application, we will just use this list: settings.INSTALLED_APPS. Using this list we will attempt to load the modules as defined in the settings.INSTALLED_APPS list with the importlib module. This module comes with a handy feature called import_module(), which takes a string (aka the string listed in the settings.INSTALLED_APPS) and attempts to import it. Once imported it returns the module object that it found.
Now is a good time for me to state that I truly love the idea that everything (and they mean everything) is an object in Python. The module object is literally an object that describes a module that has been loaded, or in this case, the app's module. From that we can get the path to the module, and using some other python tools, we can grab the name of all the other modules inside this app. This means that we can load every module in an app and never need know what the app structure looks like beforehand! Isn't that cool? Because of this, we can make use of another nifty tool: inspect.
inspect is a module in python that allows you to inspect a module, directory, or whatever it is that you may need to inspect. In this case we are going to inspect all the modules of an application that we have loaded from the settings.INSTALLED_APPS list above. It is probably a good idea to filter out the django.* apps since they wont have defined the routes as we have hear, but that is up to you.
The function from the inspect module that we are interested in is the get_members() module. This is a really neat function because you can pass in a module and a predicate (which means a declaration of need) to find what you are interested. We are interested in finding all the view classes of our modules. Doing it this way allows us to define a view wherever we like inside our application - we needn't be limited to a single views.py module. To find just the classes, do the following:
inspect.get_members(module, inspect.isclass)
This will return a list of classes. There is no way to see if a class definition descends from another class type without first creating an instance. Again, the everything-is-an-object paradigm means that the class definition is an object too, but it is simply of type type (the reason for this is far beyond the scope of this tutorial). To check if our class is of type View, we need to make an instance. Luckily the call inspect.get_members() actually returns a list of tuples, with the second position of the tuple being the class definition object. Since these objects are used to create an instance of the class, all we need to do is get the second position member and call it. An example below:
klasses = inspect.get_members(module, inspect.isclass)
inst = klasses[0][1]()
The parentheses at the end creates the object inst, which is an instance of the class that was defined by the second position of the first item in the klasses list (that is a mouthful, read it a few times to make sure you understand). We can now check that the inst class is of type view by doing the following:
isinstance(inst, View) #view must but imported before use from django.views.generic.base.View
Checking to make sure that a class is a view is necessary because now we can take the other information we have about the module, the app, and the view name and create the app/module/view route. Also, since we already have the view with its route table, we have all the information to add the custom routing that is defined on the view. Pretty sweet!
Note that it is a pretty trivial matter now to do something very similar to look for a views.py module where you can define all your functional based views and add them to the routes table as well. This makes it so that you won't have to define any routes.add() outside of your routes folder. Doing this makes it much more automagical, but it is really up to you.
Conclusion
Usually my tutorials are a lot more straightforward, but I felt like for this example it would be too long and too much to go over every aspect of the code. Also, the intent of this tutorial was to try and give some idea of how to do something versus giving just one idea of how to solve the problem. It was a lot more difficult, and so it is probably pretty unclear at times what I was attempting to do. I said I would give the source code to you to view, and so I have attached it at the bottom here. But I would encourage you to view my solutions on Github. Github has automatic syntax highlighting which makes things easier to read.
I hope that I was able to explain in some detail some of the cooler aspects of this routes creation. It took me awhile to figure it out, but now that it is done I am very proud of what it can do. I am sure that there are several people who have done this, but it seems to me that I always get more joy from figuring it out on my own. Hopefully someone can use this to their advantage.
Source
'''
Created on Mar 12, 2015
@author: derigible
'''
from django.conf.urls import url, patterns
from django.conf import settings
import importlib as il
import glob, os, sys, inspect
from django.views.generic.base import View
def check_if_list(lst):
if isinstance(lst, str):
'''
Since strings are also iterable, this is used to make sure that the iterable is a non-string. Useful to ensure
that only lists, tuples, etc. are used and that we don't have problems with strings creeping in.
'''
raise TypeError("Must be a non-string iterable: {}".format(lst))
if not (hasattr(lst, "__getitem__") or hasattr(lst, "__iter__")):
raise TypeError("Must be an iterable: {}".format(lst))
class Routes(object):
'''
A way of keeping track of routes at the view level instead of trying to define them all inside the urls.py. The hope
is to make it very straightforward and easy without having to resort to a lot of custom routing code. This will be
accomplished by writing routes to a list and ensuring each pattern is unique. It will then add any pattern mapppings
to the route for creation of named variables. An optional ROUTE_AUTO_CREATE setting can be added in project settings
that will create a route for every app/controller/view and add it to the urls.py.
'''
routes = [] #Class instance so that lazy_routes will add to the routes table without having to add from the LazyRoutes list.
acceptable_routes = ('app_module_view', 'module_view')
tracked = set() #single definitive source of all routes
def __init__(self):
'''
Initialiaze the routes object by creating a set that keeps track of all unformatted strings to ensure uniqueness.
'''
#Check if the urls.py has been loaded, and if not, then load it (for times when you want to create the urls without loading Django completely)
proj_name_urls = __name__.split('.')[0] + '.urls'
if proj_name_urls not in sys.modules:
il.import_module(proj_name_urls)
if hasattr(settings, "ROUTE_AUTO_CREATE"):
if settings.ROUTE_AUTO_CREATE == "app_module_view":
self._register_installed_apps_views(settings.INSTALLED_APPS, with_app = True)
elif settings.ROUTE_AUTO_CREATE == "module_view":
self._register_installed_apps_views(settings.INSTALLED_APPS)
else:
raise ValueError("The route_auto_create option was set in settings but option {} is not a valid option. Valid options are: {}".format(settings.route_auto_create, self.acceptable_routes))
def _register_installed_apps_views(self, apps, with_app = False):
'''
Set the routes for all of the installed apps (except the django.* installed apps). Will search through each module
in the installed app and will look for a view class. If a views.py module is found, any functions found in the
module will also be given a routing table by default. Each route will, by default, be of the value <module_name>.<view_name>.
If you are worried about view names overlapping between apps, then use the with_app flag set to true and routes
will be of the variety of <app_name>.<module_name>.<view_name>. The path after the base route will provide positional
arguments to the url class for anything between the forward slashes (ie. /). For example, say you have view inside
a module called foo, your route table would include a route as follows:
^foo/view_name/(?([^/]*)/)*
Note that view functions that are not class-based must be included in the top-level directory of an app in a file
called views.py if they are to be included. This does not make use of the Django app loader, so it is safe to put
models in files outside of the models.py, as long as those views are class-based.
Note that class-based views must also not require any parameters in the initialization of the view.
To prevent select views from not being registered in this manner, set the register_route variable on the view to False.
All functions within a views.py module are also added with this view. That means that any decorators will also have
their own views. If this is not desired behavior, then set the settings.REGISTER_VIEWS_PY_FUNCS to False.
@param apps: the INSTALLED_APPS setting in the settings for your Django app.
@param with_app: set to true if you want the app name to be included in the route
'''
def add_func(app, mod, func):
r = "{}/{}/(?:([^/])*/+)*".format(mod,func[0])
if with_app:
r = "{}/{}".format(app, r)
self.add(r, func[1], add_ending=False)
for app in settings.INSTALLED_APPS:
if 'django' != app.split('.')[0]: #only do it for non-django apps
loaded_app = il.import_module(app)
for p in glob.iglob(os.path.join(loaded_app.__path__[0], '*.py')):
mod = p.split(os.sep)[-1][:-3]#get just the module name without the .py
try:
loaded_mod = il.import_module('.' + mod, loaded_app.__package__)
for klass in inspect.getmembers(loaded_mod, inspect.isclass):
try:
inst = klass[1]()
if isinstance(inst, View):
if not hasattr(inst, 'register_route') or(hasattr(inst, 'register_route') and inst.register_route):
add_func(app, mod, klass)
if hasattr(inst, 'routes'):
self.add_view(klass[1])
except TypeError: #not a View class if init is required.
pass
if mod == "views" and (hasattr(settings, 'REGISTER_VIEWS_PY_FUNCS') and settings.REGISTER_VIEWS_PY_FUNCS):
for func in inspect.getmembers(loaded_mod, inspect.isfunction):
add_func(app, mod, func)
except ImportError:
raise TypeError("Routes type found in view module when settings.ROUTE_AUTO_CREATE has been set. Switch Routes to LazyRoutes.")
def add(self, route, func, var_mappings= None, add_ending=True, **kwargs):
'''
Add the name of the route, the value of the route as a unformatted string where the route looks like the following:
/app/{var1}/controller/{var2}
where var1 and var2 are arbitrary place-holders for the var_mappings. The var_mappings is a list of an iterable of values
that match the order of the format string passed in. If no var_mappings is passed in it is assumed that the route has no mappings
and will be left as is.
Unformatted strings must be unique. Any unformatted string that is added twice will raise an error.
To pass in a reverse url name lookup, you can use the key word 'django_url_name' in the kwargs dictionary.
@route the unformatted string for the route
@func the view function to be called
@var_mappings the list of dictionaries used to fill in the var mappings
@add_ending adds the appropriate /$ is on the ending if True. Defaults to True
@kwargs the kwargs to be passed into the urls function
'''
self._check_if_format_exists(route)
def add_url(pattern, pmap, ending, opts):
url_route = '^{}{}'.format(pattern.format(*pmap), '/$' if ending else '')
if "django_url_name" in opts:
url_obj = url(url_route, func, kwargs, name=kwargs['django_url_name'])
else:
url_obj = url(url_route, func, kwargs)
self.routes.append(url_obj)
if var_mappings:
for mapr in var_mappings:
check_if_list(mapr)
add_url(route, mapr, add_ending, kwargs)
else:
add_url(route, [], add_ending, kwargs)
def add_list(self, routes, func, prefix=None, **kwargs):
'''
Convenience method to add a list of routes for a func. You may pass in a prefix to add to each
pattern. For example, each url needs the word workload prefixed to the url to make: workload/<pattern>.
Note that the prefix should have no trailing slash.
A route table is a dictionary after the following fashion:
{
"pattern" : <pattern>',
"map" :[('<regex_pattern>',), ...],
"kwargs" : dict
}
@routes the list of routes
@func the function to be called
@prefix the prefix to attach to the route pattern
'''
check_if_list(routes)
for route in routes:
if 'kwargs' in route:
if type(route['kwargs']) != dict:
raise TypeError("Must pass in a dictionary for kwargs.")
for k, v in route["kwargs"].items():
kwargs[k] = v
self.add(route["pattern"] if prefix is None else '{}/{}'.format(prefix, route["pattern"]),
func, var_mappings = route.get("map", []), **kwargs)
@property
def urls(self):
'''
Get the urls from the Routes object. This a patterns object.
'''
return patterns(r'',*self.routes)
def _check_if_format_exists(self, route):
'''
Checks if the unformatted route already exists.
@route the unformatted route being added.
'''
if route in self.tracked:
raise ValueError("Cannot have duplicates of unformatted routes: {} already exists.".format(route))
else:
self.tracked.add(route)
def add_view(self, view, **kwargs):
'''
Add a class-based view to the routes table. A view that is added to the routes table must define the routes table; ie:
(
{"pattern" : <pattern>',
"map" :[('<regex_pattern>',), ...],
"kwargs" : dict
},
...
)
Kwargs can be ommitted if not necessary.
Optionally, if the view should have a prefix, then define the variable prefix as a string; ie
prefix = 'workload'
or
prefix = 'workload/create
Note that the prefix should have no trailing slash.
'''
if not hasattr(view, 'routes'):
raise AttributeError("routes variable not defined on view {}".format(view.__name__))
if hasattr(view, 'prefix'):
prefix = view.prefix
else:
prefix = None
self.add_list(view.routes, view.as_view(), prefix = prefix, **kwargs)
class LazyRoutes(Routes):
'''
A lazy implementation of routes. This means that LazyRoutes won't add routes to the Routes table until after the
routes table has been created. This is necessary when the ROUTE_AUTO_CREATE setting is added to the Django settings.py.
All defined routes using the routes.* method must now become lazy_routes.* methods.
'''
def __init__(self):
'''
Do nothing, just overriding the base __init__ to prevent the initilization there.
'''
pass
lazy_routes = LazyRoutes()
routes = Routes()
