CVGM.net - Chiptune, Retro & Video Game Music

Hi,

From now on, all site changes & code updates will be posted in this single thread as a single reference thats easily found by everyone. Adding these updates in individual threads was a bit confusing at times.

Once again, Terrasque has been working hard on the next set of site changes. Internally, he has been making lots of changes to the caching system, where the site will now cache the queries from the site pages, rather than the data thats displayed on the site. He has also re-written the smiley parser so it is now far faster than before. Both of these changes have increased the speed of the site dramatically.

To celebrate the occasion, I restored the old smiley database and the site is still lightning fast

Some of the additional changes that have been included:

* Player Preview - This has been moved to a window of it's own. It is still sharing with the new Java player that we are experimenting with, and it no longer clutters up any of the view screens (especially for moderators).

* Many templates have been streamlined, which makes the code look less crap and also takes advantage of other features in the code base.

* Some work on translation support has begun, though the effects won't be seen until a future update.

* Smiley spam filter. Site owners can now limit the number of total smileys per message, and the number of times a single smiley can be used in that message (to stop a flood of the same smiley being used)

* Fixed some bugs with non-english characters being displayed incorrectly in song title and artist lists.

* Compilations can now be edited by staff and use the same drag/drop ordering on songs. This is still being tested and so might not be available to everyone yet.

As always, be sure to thank Terra for the great changes, and if you have any bugs or problems to report with the changes, you may do so here in this thread. Thanks

Hi,

I have updated the search code on the site to the latest version today. The old search has been removed, and replaced with what was the Beta search before. This see's some great improvements to the amount of information found when searching on the site.

In addition to this, I have now set up automatic polling of the search data to a 15 minute interval, so if something was just recently added to the site it might not show up on the site for up to 15 minutes (probbably only most notable on areas like compilations, but you can still use the SongID as the search term).

I will update the internal Whoosh system later in the week, as some updates have been added to the tool that will make search results a little faster. If anyone has any questions, feel free to ask. Enjoy!

Hi,

I have applied several major changes to the site today. Firstly, I addressed some issues regarding other sites using our data in theirs, and making it look like their own. This now shouldn't be a problem, and people won't be fooled into thinking that it belongs to someone else It will also stop Google posting results for everywhere but our site

Additionally, I have removed the slacking gloves and updated many other things. Firstly, I caught up on the 8+ revisions of the web server we use, so that is now up to date with many new speed & internal cache improvements. The box that CVGM runs on has also been brought up to date with the latest Linux kernels and internal software.

The site software has also been upgraded. The search engine has been upgraded to the very latest versions of Whoosh and Haystack, and the other python/django pieces that go with it. Also upgraded were the internal django setups, new search indexes built and everything else. You will immediately notice all search operations are now considerably quicker than they were before!

I have also updated the statistics section of the site, adding 2 new statistic features (Random and Least Votes). Random picks 100 random songs from the database, so if you get bored you can just queue some of the random findings! Least Votes show the tracks on the site which have the lowest vote counts, so you can listen to them and place your votes.

As always, if anyone has any questions feel free to ask FishGuy876

Hi,

There have been a few site changes over the last few weeks, this post will try and go over some of the massive changes that have occured so you can familiarize yourself with them.

CVGM Has now moved to a fully 64bit linux installation. This has remedied a few problems we were having with the site, the software it uses, and server load. Now that the site is becomming more popular to listeners, we needed to upgrade so that we can keep up with the demand. We don't expect that we will need any more major upgrades in the future, but you never know! Thanks to Rams for donating storage space to use during the migration process.

Screenshots - One of the biggest visual changes to the site is the use of Screenshots. This system is still under development, and I have mnade a seperate forum thread to talk about it. To learn more about the screenshot system, please read Working With Screenshots On CVGM. This will also explain how to add screenshots to songs, navigate existing ones and more.

Compilations - The way that compilations are created & edited has now been altered. It is now possible for any registered user on the site to creat a compilation using any songs in the database. To create a new one, simply browse to the main "Compilations" page and click "New Compilation". A Screen is presented with the usual fields for creating the compilation setup. The new part is at the very top, you can search for songs by name, or typing in their SongID numbers. Once songs are found, you can drag them into position underneath, even controlling the order that they appear in the compilation.

Once your compilation has been created, you should let a member of staff know that its waiting as it won't be active until they approve it. Staff members can edit any existing compilation on the site by appending /edit to the end of it's URL. A more detailed page for creating compilations will appear soon.

User/Site stats - Visit any profile of any user and you can now see some stats about them and the site, see how many times they have chatted in the oneliner, how many tracks they have uploaded and also how many votes they have cast on songs. There is also a new "Recent Updates" section of the site, which displays the most recent changes to all Groups, Artists, Labels, Songs & more. See at a glance what is being changed on the site, newly added artists etc.

Better OpenID support - If you have a compatible OpenID account with another site or service, you can now use it to log in to CVGM. You can now log in using a Google, Yahoo, AOL, OpenID, LiveJournal, Wordpress ID and many many more! Try it today!

Song Licenses - You can now assign licenses to songs, and we have started off by adding the most common Creative Commons licenses to the database. These licenses control how songs can be edited and distributed.

As always, any other major updates we make to the site, we will keep you in touch. Thanks!

Hi,

Updated the codebase a little bit tonight. Terra has fixed some issues in the XML system, I will paste his message from http://www.scenemusic.net/forum/thread/387/ here. If you see of people reporting issues with remote clients, this is most likely why :



Other minor changes:

* Screenshot fix if pouet prod alrady exist in image db.
* Improvements to Tab Completion.
* Unicode strings in dv_extend, and several other internal changes.

Thanks!

Hi,

Over the next few days I am going to implement a few changes that may affect the site. Firstly, our DNS provider (EveryDNS.net) will be moving their system over to DynIP, so I must also move over my settings to their new servers. Hopefully, this won't cause any downtime to the site.

Secondly, I am going to be migrating the streamer to an older version to get things back in syncronization with each other. This older version has a few known problems, and from time to time when it crashes it will eat some of the queue entries. Unfortunately, this will have to do until the newer stream versions are fixed.

As always, if anyone has any questions, feel free to get in touch with me via a PM, or use this forum. Thanks!

I have made the changes to the streamer, and so far they seem to be working quite well. We have gone back to an old BASS-powered version of the stream, which cuts down on the delay between the site and the stream.

This new version does add some other problems, as always Firstly, this version carries the queue-eater bug where if the streamer crashes, it munches away at some of the queue entries before it restarts again. Secondly, if you listen to the stream for long periods of time, it slowly gets out of whack as songs change. After 4 hours of listening, you may be off by about 20 seconds or so. Simply restarting the stream fixes this issue.

In the next day or so, I will begin to switch the DNS. I am also having to downgrade the server that CVGM is running on to ensure I can keep paying for it; Situations at home have changed a bit and I cant afford to splash out on an expensive server for the time being. The last thing I need is a lapse of CVGM being offline for a month or two =/ If you know anyone with a spare million bucks, now would be a good time to donate it

As these changes begin/commence, I will keep you posted with whats going on. Once these changes are done, I am going to start recoding some bits of the site. Stay tuned!

Its been a bit of time, but I have applied some new code changes to the site, mostly bugfixes that Terra has been working on from a while ago. There are not many major new features, except for a theme browser, which I need to set up properly and link to it later. If anyone has any issues on the site, let me know. Thanks.

Its about time that this thread was re-opened

As of October 2012, CVGM has moved the code repo to SVN on Google Code. The previous repo was hosted by Terrasque on Gitorious using Git, and several people didn't like to use the tree very much as they found Git a bit overpowering. I myself don't like to use Git, which is why I have opted for an SVN repo instead.

All future bugs and issues for this site should be posted on the Google Project page above, or here in the forums. If you do find a bug on the site, Check It Hasn't Been Reported Already before making another ticket. You can also look at the code exactly as it runs here and make suggestions and changes for other bugfixes.

Small Changes For Today:

* Stats/Voted - Songs with the highest overall rating are shown at the top of the list, they are now sorted also by the number of votes cast, so high voting songs with high vote counts (rather than a 5 star song with only 1 vote) will display at the top of the page.

* Fixed a template bug where a song didn't show which songs had been remixed based on itself.

* The 'Who's Online' section now shows X most recent members, where X is a value in settings_local.py

Other fixes over the last few days:

* Added a change that lets site admins turn off the email notifications when new
song info is approved. The system will still email a user if their request is
denied.

* Added additional flags. There should be no invader flags for certain countries anymore.

* Improved Thumbnails - Site now makes better, high quality thumbnails of screenshots. Site admin can also alter the output format of the thumb (between JPG and PNG) and a few other settings such as quality. Low colour screenshots (such as Spectrum) now convert at a much better quality then before. See the below comparison of thumbnail images:

Original Image (Unaltered):


Previous Thumbnail Attempts:


New Thumbnail Attempt:

Hi,

More big changes \o/

The first big change on the list is the streamer. The site is now using the most recent version of maep's Demosauce streamer. The version we used before was at least 2 years old, the new one fixes countless bugs, has better system performance, and puts the stream in far better sync than ever before. There are also several internal systems on each song page's admin panel that can now be utilized with the new code, such as picking a BASS replay method for specific mod songs, and a few others. If any bugs or problems crop up with the streamer, let me know.

All the missing flags from the new country box have now been added. I dont think there are any left that show the Invader flag anymore.

The Voted stats page (here) now sorts tracks by the best rating. Songs with the best ratings are sorted to the top. Songs with a max rating of 5 will show by order of the most votes cast first, making more popular voted songs appear on top.

The New Members section of the site now shows the last 5 members to join the site. Don't be afraid to say Hi and welcome them to the site when you see them online

Maximum length for a song name has been extended to 128 characters, and filenames can be a maximum of 256 characters. Filenames should never be that long anyways, but it is there just in case.

I have altered the thumbnail/screenshot section again. I have added a couple of minor optimizations which will only be effective on a handful of images and image types, but the outcome will still be clean crisp thumbnails depending on the source image.

I have also added the ability for staff members to alter the current thumbnail for any screenshot. There are many thumbnails on the site that just look crappy due to the old scaling methods, such as GIF images scaled with very low colour counts.

All that the staff member needs to do is go to the screenshot page. Staff members now see an additional pane related to the thumbnail for the screenshot. After the text is a link named "Rebuild Thumbnail" which will rebuild the thumbnail based on the current image, using the settings for images defined for the site (output format and quality settings, CVGM uses PNG with a quality setting of 90). If you are currently using a custom thumbnail, it will be erased, and the scaling does not take into effect for AnimGIF files (you can still do those manually in the Admin panel). Below is a screenshot showing the effect, and what the panel looks like. The thumbnail on the Left is the old, crappy GIF that was present. The thumbnail on the right (thats merged into the image) shows the new thumbnail after rebuilding. Much better quality

Another Before/After example of thumbnail cleansing:








I'll stop showing examples or my OCD will have me update the entire screenshots section

Thumbnail feature is totally awesum.

It's so good, that - actually - is there any reason to keep old manual thumbnail upload on the screenshot form?

Thanks!
-gF

The thumbnail portion of the Admin form is put there automatically, when a new screenshot is uploaded the thumbnail is generated automatically and added there. Removing it isnt required, especially as some screens may want custom thumbs (such as anim gifs)

I still use it for animated gifs

I wonder how it will work with interlaced gif images, like in Red Alert or Gabriel Knight 2 for example

Gotta test it. It really peeked my curiosity

Hi,

I have recently applied a couple of large patches to the site full of changes, so this post will go over some of the more important ones, so you know what to expect when things are different!

Cross-browser sorting of songs by queued status - When viewing your favourites list, instead of sorting by the last request date, you can now sort by the Locked/Unlocked status as well. Patch by debo.

Linking Only To Active Screenshots - When adding/linking to existing screenshots, only ones that are Active will now work. Pending screenshots must first be approved before they can be used. Patch by debo.

Recent Forum Posts - When you visit the Forums page, you will now see a list on top of the most recent forum posts. These will also show if they were posted in the same subject, so you can tell if a particular thread has been busy.

Anonymous Screenshots - Users who are not logged into the site, such as those just having a peek around for the first time, will now be able to see screenshots for songs that are playing. The old code didn't allow this, so I added this patch (which is controlled via a setting) so the site looks a lot nicer to others. It will also show the average vote rating for songs underneath the title, just like you were logged in. Obviously, you cannot place votes unless you are logged in.

Browse Songs By Year - View all the songs that are released in the given year. You can view all songs for say, 1989 by going to http://www.cvgm.net/demovibes/songs/year/1989/. It also works when you view the Song Details of any song in the database, and clicking it's year (which is now a hyperlink). Invalid years will result in a blank results page. All results are paginated, and can be navigated very easily.

Oneliner Muting - See below for a more detailed description of this.

Other smaller bugfixes & changes:

* Fixed the titling of several statistics pages, so they now have a meaning ful title in the tab/window.
* Adjustment of Registration/Password Reset forms to be a little more clear on their instructions.
* Fixed the bug where all 5 star rating counts were not shown in overall stats.
* The vote counts for each rating were backwards (0 was actually 5 stars, 5 was actually 1 stars etc.) so this has been corrected to show the right number for the right vote count.
* Cleaned up the templates for the Overview statistics for the station. Better wording & clarity on some areas.
* Site can now globally disable all uploads with a simple setting change. This will mainly be used only during maintenance periods. In the past, the upload forms had to be disabled manually, which didn't always work.
* When adding a new compilation, the number of song search results is now controlled by a setting, so they can be increased/decreased accordingly.
* Several other template tweaks & instruction/help text alterations.
* Support for multi-artist uploading will NOT be added in the near future.

Several internal code changes have been made, using patches originally done by Terrasque and modified by me for use on our code tree:

* Made adjustments to the internal event system, so its not always trying to do everything immediately. It also helps to balance out load when a lot of stuff is going on.
* Fixes to the Event Handler system, JavaScript idle user changes, again all helping to reduce site load.
* OneLiner History Searching.
* Per-User Smileys, & Smiley Improvements to increase speed & handling internally.

Oneliner Muting

In the past, we have had some issues with users that just needed to be banned for various reasons, or sometimes people need a warning when they are being offensive to others. The site now has Oneliner Mutes, based on a code patch by Terra (my original patch worked quite different, and this patch is actually a lot better than mine). In the event a user is getting out of hand, any member of staff now have the ability to mute that user (which prevents them from speaking in the oneliner, or doing anything else on the site for that matter) for a specified period of time. The default is 15 minutes, however depending on the event, a user can be banned for up to 7 days.

There are also measures to prevent users with many alts getting around this, as the site (optionally) will ban the IP address the user had logged in with. This will prevent the main user logging out, and logging in with an alt.

When a ban is posted, a notification will appear on the top of the screen like any other (unless you have notifications turned off). Staff members can see a list of all active bans, and ones that have been issued in the past.

As always, if you have any questions about this set of patches, or anything else with the site, feel free to post a reply here, or start a new thread, or contact me directly. If you find bugs on the site that need fixing, or you want to request a new feature, I reccomend you add a new Issue to our code tracker over at https://code.google.com/p/demovibes-cvgm/issues/list. This also shows the currently Active/Inactive issues we are dealing with. Thanks!

FishGuy876

Something I forgot to add was a change to the OneLiner text box. Depending on your browser, you may need to empty the cache if you don't see this change appear.

The text that you normally click under the box to show smilies has been altered. Now you will see a "Link:" tag, followed by a "Page" and/or "Smilies".

Clicking Smilies will naturally take you to the list of available oneliner smilies, which you can click and it will be instered into the text box.

The "Page" option appears when you are viewing any forum, song, screenshot or other detail page. When you click the Page button, it inserts the bbcode for the object in view. If its a song detail page, it'll insert the [ song ]SongID[ /song ] tags for you (SongID being the ID number of the song for that page.

Hey Everyone,

Just a small post about some site changes I have been working on. My main job is calming down a bit, so I am hoping to get more time devoted to working on new changes and features for CVGM. I know I have been slacking for quite some time, I recently set up a new dev server and have been tinkering around with a few things.

I added a change to the way that Favourites are listed. In the past, I always had the same bunch of favourites up on the top few pages because I didnt want to play them, or they were too long. My favourites list is also a couple of thousand entries long now

The new change re-arranges the one favourites column (now titled Last Played). When I pull up the list of my favourites, this column now shows the date when the song was last played (actually played). Previously, it showed when it was last requested. If you sort by this column (as I do) then the songs that have not played in a long time will appear at the top, and recently played stuff at the bottom. If you click the header again it will sort the list the other way.

I wanted to change this to refresh favourites that haven't played in a long time, so if the song plays (even if it's by DJ Random) it will change order and hopefully those stale lists of favourites on top of everyones pages should start to disapear.

If you have other ideas and feature requests for the site, feel free to post about them in the correct areas and I will see what I can do to add them. I have started going through old posts and making some lists of things to work on, and always like to listen to new ideas

FishGuy876

Hi All,

I made a small change to the streamer code this week and was documenting it here before I forgot about it. The actual commit details can be found on GitHub Here.

What was happening when a large song/set was being uploaded is the scan tool was timing out after a few seconds. At the time, CVGM was using V0.4.0 of the old scan tool codebase (compiled long ago in a galaxy far far away) as I had an old not to update from eons ago and just never really got around to it.

Anyways, the limit on the length of the song to scan is contained in the streamer source code, so I applied a change to the length and recompiled both the streamer, and the scan tools to reflect this change. I also modified the CVGM code to work with this newer version of the scan tool, so as we recompile in the future there should be no more problems.

I do beleive that there is a new version of Conrad's streamer, but I cant say for sure how compatible it is with the code and I will assign myself a task to look at that in the future. Also, I have put an issue in the bugtracker to modify the current code to work from a setting within demosauce.conf instead of having to recompile the app every time we want to make a change.

As always, should anyone have any issues or problems please let me know. Thanks!
FishGuy876/BRS

Just a quick update on the streamer update. I found a potential problem with a few mp3 files, and even though they were just 3 or 4 minutes long, the were still failing with length errors. I applied a temporary fix to the code at CVGM, and I added a comment to dscan.c file (See the commit Over At GitHub).

I ran some testing on a few mp3 files and there appears to be no problems caused by commenting out the time checking; If the file is somehow messed up, it will be flagged as an invalid format before it causes any major issues anyways. Thanks!

FishGuy876

Hi Everyone!

I have been doing some work on the streamer code again this week. The last changes I made to help with the longer songs being uploaded were not the best or the most stable, so I have been working on a more stable solution to the problem.

Firstly, I have upgraded the streamer code to the most recent version from maep's repository (which can be found at https://gitlab.com/maep/demosauce for those interested). It's not on Github, so I created a new branch on the demovibes-cvgm repo just to handle work on the streamer. By updating the streamer, it was very obvious to me that the following fixes have already occured:

- The most recent code is being used, with OpenMPT and all the other bells and whistles in maep's repo.
- Scanning of long songs is now working as it should; We will need to fix a couple of issues on that anyways as we progress.
- Demosauce no longer crashing when longer songs are being played, after the song has finished.
- BUG: If the dscan and demosauce tools cannot find the libbass.so shared file, manually copy it to /usr/lib/ manually to fix the issue. The old tools used to create a symlink to the files locally. Its located in the lib folder and is automatically downloaded during the make process.

After building the new tools, I have applied the following fixes to CVGM:

- There should be no more issues with uploading files and a 500 error due to the format being unrecognized. If you have an issue, send me a PM or post a comment here with a link to the source file so I can see whats going on.
- As a side effect of the new scan tool, scans do take longer to complete (the 1.5 hr lug00ber song takes about 15-20 seconds after uploading) but im OK with the extra time as long as it works correctly.
- To combat the extra time, the initial scan will calculate the replaygain values at the same time. In the old code, this would scan a 2nd time to retreive this value. Now, we will only scan for just ReplayGain if it's set to a NULL value in the Admin Panel.
- I have currently disabled maximum song length uploads. See below in the fixes for some extra notes on this topic.
- Link to the Streamer branch can be found here: https://github.com/FishGuy876/demovibes-cvgm/tree/demovibes-streamer

Stuff to fix in the future:

- I want to go through all of the code and just make sure that all the features are being used, and being used correctly. I might even re-write parts of the dscan.py file in the future to optimize it, or just wait until a django3 version of cvgm is in testing properly.
- I want to move the fixed in-code maximum length of a song to being a value in the settings file. CVGM doesn't really care too much about the maximum length at this point, but for other sites that use the code it may be beneficial to them to have such a limitation. The code also needs to recognize this and handle it correctly instead of throwing a 500 error.
- I want to modify the dscan tool to properly log whats being uploaded/scanned etc. to better help with troubleshooting in the future.

I want to extend a huge thanks to maep for his continued work on the streamer, and to everyone else at CVGM for their patience. Thanks guys!

FishGuy876/BRS

Thanks for keeping CVGM wheels rolling, Fishy.
You are The Man Master Fishtainer!

\o.