Re-write of collaboration layers: infos about current design & functionality
Last saved by Pierre henri Seylan on December 2, 2014
Here are a few informations about specific parts of the current collaboration modules in the Vapor app.
Some of them don't need to be re-written now and could be used in the new structure.
Also, some modules were not completed and need discussion about how to best implement them, additional informations will be shared in separate documents for those modules.
Finally, you will find a list about the consequences of the current design, those must be taken into consideration with the new design, so that we don't bump into the same problems again.
1.Code to re-used and document more consistently:
1.1 Feeds composing and merging:
Feeds must be composed before to be sent on the server, this step happens locally.
A feed composition means, taking a part of the value tree project and packing it into a zip file.
The important part to be careful with is the mechanism that takes care of selecting and extracting the selected parts of a project to compose a feed.
As well as when you receive a feed, this one needs to be merged into your project, a mechanism is also in placed to be able to integrate/merge a feed into an existing project.
I'm talking here about the locally aspect of it, once the feed has been downloaded from the server, the mechanism described here is the one that takes care about the local merge into your project.
1.2 Streaming files from the server:
One feature that is not to re-implement is the possibility to pre-listen files audio from the server.
This was implemented only for projects, but was working currently.
You should be able to find the streaming code in the feature for pre-listening projects.
The streaming code will be used to pre-listen media library file as well as projects and feeds, it is an important part...
1.3 Background tasks management:
The application is able to process tasks in the background, those tasks are divided into audio and server tasks, both are performed on a background task thread pool.
It is important to pay attention to this background task management, since we will need to replaced the existing server tasks without to break the management of the audio tasks.
In the Backgroundtaskqueue class it can be seen how the server and audio tasks are handled.
1.4 List of things that must remain possible in new design:
2.Uncompleted modules:
2.1. Media library:
The media library was never completed properly, it was only to the point were you could download a server file, here is a rough list of what is missing:
2.2 Notifications:
There was never a proper notification mechanism, and implementing one with the current design would have been a sort of work around.
We need a proper notification system, that integrates well with the website notifications.
3. Visible consequences from the current design:
3.1 Difficulty to extend the code:
It is currently very hard to add new features, but it is very easy to break existing ones...
This can be due to but not only:
3.2 Everything packed into single zip files:
One problem is that when you push files, they are packed into a single zip file, for example, a project of 1GB would be pushed into a single file, which is critical.
We should make a design where it is possible to push the different parts of a project separately.
The question is how much changes will be needed on the server?
3.3 No possibility to cancel calls:
It seems that it was impossible to cancel a call, for example when pushing a project you couldn't cancel the task.
Tuschl told me that this was due to the fact that the calls are made all synchronous.
In the new version, it should be possible to cancel a task in progress.
3.4 All the functionality are project based:
Somehow, the design was so static that you everything was based around a project, for example, it would be impossible to download and feed from project A and merge it in project B, no flexibility...
Cloud project could not be opened at the same time, and only 1 cloud project could be opened at a time, even if the handling of multiple project window is handled by JUCE, it had to be disable for server project because of the design.
3.5 Local projects could not be pushed:
"Pure local" project could not be pushed to the server, because they didn't belong to any server project.
It should be possible to push a local project at any time, and simply choose to create a new server project, or to assign the project to an existing project you have on the server by pushing it as a new private branch.
3.6 No Offline mode:
When the server is down, or when you don't have access to the web, you simply couldn't open the app, no offline mode was allowed.
We should have a better approach for this and try to make something like spotify.
The user password could be saved in the keychain, and you could access your local projects even if the app is not online.
When trying to use server features, you would be requested to login.
4. Server side:
There probably many more bottlenecks on the server, however here are a couple of problems we have currently:
4.1 Permissions and files access:
Currently all the permissions are handled locally, e.g the client will block some actions depending of the user role in a project.
We must change this approach and have something like a "Read/Write" system, where a user would have read or write access to a file.
This approach seems to be flexible and could be used for projects and media library.
4.2 Interaction with the server data base:
On the current implementation, the server blindly execute all the actions that are requested by the client, without to keep track of it by writing datas into the DB.
This has the consequence that we can't keep track of what has been done in a project, for example having a history of the file posted by various users in a project, if files are deleted or replaced the system is not able to track those changes.
We must implement a module on the server which interacts with the DB and keeps track of projects and media library activities.
At first sight, all the infos that must tracked into the DB are:
Projects:
Media library:
4.3 Files backup
The above mentioned prevents us from allowing files recovery.
When you push a file, for example a new version of the master, the previous file is simply overwritten.
This behavior is a problem for collaborations, since in a project, the master version could be modified a lot over time, the users may want to recover previous versions of the project.
By keeping track of changes in the DB, we should be able to implement a "version history" feature.
Also, it seems that this functionality is already supported in Amazon S3, but our server DB must be capable of handling this functionality.
Some of them don't need to be re-written now and could be used in the new structure.
Also, some modules were not completed and need discussion about how to best implement them, additional informations will be shared in separate documents for those modules.
Finally, you will find a list about the consequences of the current design, those must be taken into consideration with the new design, so that we don't bump into the same problems again.
1.Code to re-used and document more consistently:
1.1 Feeds composing and merging:
Feeds must be composed before to be sent on the server, this step happens locally.
A feed composition means, taking a part of the value tree project and packing it into a zip file.
The important part to be careful with is the mechanism that takes care of selecting and extracting the selected parts of a project to compose a feed.
As well as when you receive a feed, this one needs to be merged into your project, a mechanism is also in placed to be able to integrate/merge a feed into an existing project.
I'm talking here about the locally aspect of it, once the feed has been downloaded from the server, the mechanism described here is the one that takes care about the local merge into your project.
1.2 Streaming files from the server:
One feature that is not to re-implement is the possibility to pre-listen files audio from the server.
This was implemented only for projects, but was working currently.
You should be able to find the streaming code in the feature for pre-listening projects.
The streaming code will be used to pre-listen media library file as well as projects and feeds, it is an important part...
1.3 Background tasks management:
The application is able to process tasks in the background, those tasks are divided into audio and server tasks, both are performed on a background task thread pool.
It is important to pay attention to this background task management, since we will need to replaced the existing server tasks without to break the management of the audio tasks.
In the Backgroundtaskqueue class it can be seen how the server and audio tasks are handled.
1.4 List of things that must remain possible in new design:
- The app can handle multiple server tasks on different threads, i.e pulling files while pre-listening an server audio file.
- The app can communicate directly with the amazon S3 server
2.Uncompleted modules:
2.1. Media library:
The media library was never completed properly, it was only to the point were you could download a server file, here is a rough list of what is missing:
- Nested folder structure on server side
- Tracking the library structure in the site DB
- Ability to have files tags, for example genres: rock, hip hop
- rating of files
- Browsing the list of sounds using tag filters
2.2 Notifications:
There was never a proper notification mechanism, and implementing one with the current design would have been a sort of work around.
We need a proper notification system, that integrates well with the website notifications.
3. Visible consequences from the current design:
3.1 Difficulty to extend the code:
It is currently very hard to add new features, but it is very easy to break existing ones...
This can be due to but not only:
- Inconsistent naming of classes and functions
- Very large classes containing to many functionality in one place
- too much custom code that should be contained into a library
3.2 Everything packed into single zip files:
One problem is that when you push files, they are packed into a single zip file, for example, a project of 1GB would be pushed into a single file, which is critical.
We should make a design where it is possible to push the different parts of a project separately.
The question is how much changes will be needed on the server?
3.3 No possibility to cancel calls:
It seems that it was impossible to cancel a call, for example when pushing a project you couldn't cancel the task.
Tuschl told me that this was due to the fact that the calls are made all synchronous.
In the new version, it should be possible to cancel a task in progress.
3.4 All the functionality are project based:
Somehow, the design was so static that you everything was based around a project, for example, it would be impossible to download and feed from project A and merge it in project B, no flexibility...
Cloud project could not be opened at the same time, and only 1 cloud project could be opened at a time, even if the handling of multiple project window is handled by JUCE, it had to be disable for server project because of the design.
3.5 Local projects could not be pushed:
"Pure local" project could not be pushed to the server, because they didn't belong to any server project.
It should be possible to push a local project at any time, and simply choose to create a new server project, or to assign the project to an existing project you have on the server by pushing it as a new private branch.
3.6 No Offline mode:
When the server is down, or when you don't have access to the web, you simply couldn't open the app, no offline mode was allowed.
We should have a better approach for this and try to make something like spotify.
The user password could be saved in the keychain, and you could access your local projects even if the app is not online.
When trying to use server features, you would be requested to login.
4. Server side:
There probably many more bottlenecks on the server, however here are a couple of problems we have currently:
4.1 Permissions and files access:
Currently all the permissions are handled locally, e.g the client will block some actions depending of the user role in a project.
We must change this approach and have something like a "Read/Write" system, where a user would have read or write access to a file.
This approach seems to be flexible and could be used for projects and media library.
4.2 Interaction with the server data base:
On the current implementation, the server blindly execute all the actions that are requested by the client, without to keep track of it by writing datas into the DB.
This has the consequence that we can't keep track of what has been done in a project, for example having a history of the file posted by various users in a project, if files are deleted or replaced the system is not able to track those changes.
We must implement a module on the server which interacts with the DB and keeps track of projects and media library activities.
At first sight, all the infos that must tracked into the DB are:
Projects:
- project creation
- all project push and pulls
- all feed push, pulls and merges
Media library:
- pushing and pulling of media library files
- sharing of files
- rating of files
4.3 Files backup
The above mentioned prevents us from allowing files recovery.
When you push a file, for example a new version of the master, the previous file is simply overwritten.
This behavior is a problem for collaborations, since in a project, the master version could be modified a lot over time, the users may want to recover previous versions of the project.
By keeping track of changes in the DB, we should be able to implement a "version history" feature.
Also, it seems that this functionality is already supported in Amazon S3, but our server DB must be capable of handling this functionality.