Upcomming AutoLoader pluggin

[allpowerful32]

Hi all,
I'm currently working on a pluggin called iAutoLoader, which will serve as a full threaded load pluggin, allowing the main thread to check up on the progress of the load (the overall progress, and the file progress).  I looked through the current pluggins for CS, and it seemed that a class like this one is sorely needed.  The user would add several files to the load stack, via the AddToLoad function, and then call the StartLoad function to start the load of those objects in a new thread.  I am planning to submit this code back to the community, which brings me to my question:  are there any comments, questions, or requests for the system?  FYI, I am hoping to release it before March of this year (I am still in school, and can't devote all of my time to this project.)

Thanks in advance

[allpowerful32]

I've gathered my thoughts and it seems that I will be copy-pasting most of the loading functions from the csLoader class and inserting calls in them periodically to update the progress.  In effect, I will be re-doing the csLoader class with a few extra features.  Please vote on the above poll.  Any ideas and suggestions are welcome.

[jorrit]

I've gathered my thoughts and it seems that I will be copy-pasting most of the loading functions from the csLoader class and inserting calls in them periodically to update the progress.  In effect, I will be re-doing the csLoader class with a few extra features.  Please vote on the above poll.  Any ideas and suggestions are welcome.

You are aware that the CS engine is not thread-safe? How were you planning to solve this issue? We lately discussed this too and in the end we came to the conclusion that it would be better not to make a full threaded loader but instead allow mesh object loaders to support streaming vertex and triangle data to a faster representation then the XML. That's why iStreamSource was added. Every loader plugin gets a pointer to such a stream source and can use that to fetch vertex and triangle data from. This is currently not implemented though. The API is there but that's all. I personally think it would be better to pursue this route. If you are interested in this then I can help explain how it all is supposed to work.

Greetings,

[allpowerful32]

If i'm not mistaken, what is not thread safe about CS is the fact that it has no barriers for the both threads to try to access (look at or change) information in a certain part of the app.  The iAutoLoader would be given some form of callback to call when it is ok to access a certain resource.  Instead of having a mesh object just displayed on the screen as it is loaded, have it inserted into a buffer located within the engine (or related resource manager class).  The main thread would then have to call some sort of "ReadyResource()" function in the engine class that would move the resource from the buffer into the real mangaer class.

[allpowerful32]

Also, if I decided not to make this iAutoLoader pluggin, I would make something similar for myself located in my executable as opposed to in a plugin.  I decided that I might-as-well make it available for scruiteny, and (if desired) usage by the CS community.  I don't like it when a game simply displays an unchanging image because I never know whether its locked up or actually doing something.  I am going to fix that problem one way or another.

That's why iStreamSource was added. Every loader plugin gets a pointer to such a stream source and can use that to fetch vertex and triangle data from. This is currently not implemented though. The API is there but that's all. I personally think it would be better to pursue this route. If you are interested in this then I can help explain how it all is supposed to work.

As for why I don't want to use the iStreamSource class: it doesn't seem to have any support whatsoever for progress checking, so there is still no way to tell if the game is locked up (it could be that just the loading thread is locked up). 

This class does seem like a good candidate for the buffers in the manager classes, though.

[allpowerful32]

Progress update:
The first *working* version of the plugin is within weeks.  For this version, I will scale down the total project so that the thread will only work on loading the file into an iDataBuffer object within iAutoLoader.  The main thread will then call a finalize function in the iAutoLoaded class that will call the various iLoader functions for loading from a buffer.  The amount of time it takes the iLoader plugin to move and modify the data in memory should be almost negligable compared to the time required to load the resource from disk, so I will still be accomplishing my goal. 

Right now, the header is mostly finished and the cpp file is about a third finished.  I still haven't done any testing, and I expect that to take a fair while.

[jorrit]

Nice! Keep us posted on progress

Greetings,

[allpowerful32]

Im about a week away from releasing a preliminary version.  For this, I have 2 questions: where do I put the interface header file within the project hierarchy(sp?) and also where should I put the source so that others can view it (and it can be included in a CS release)?

[jorrit]

Im about a week away from releasing a preliminary version.  For this, I have 2 questions: where do I put the interface header file within the project hierarchy(sp?) and also where should I put the source so that others can view it (and it can be included in a CS release)?

I don't know yet about where to place it. Does it belong with the map loader? In that case in include/imap. Anyway, when you are ready just send me all modified files. First make sure that you are using the latest CVS of CS so that it is easier for me to apply the changes. I will see what you did and if I like it it will be included with CS.

Greetings,

[allpowerful32]

It seems that, however hard I try, I can't get past a fundemental chain or errors - I fix one, and another turns out to be "hiding" behind it.  So far, I havent been able to get any of my code de-errored, as the problem code always seems to be in some CS header file.  I am simply trying to get to my errors, which I tend to be fairly good at.  I am not good at fixing compatibility errors with a system that I didn't design.  Here is what I have so far:

In CS\Plugins\autoloader\csAutoloader.cpp:

Code:

/*
    Copyright (C) 2006 by Joshua Warner

    This library is free software; you can redistribute it and/or
    modify it under the terms of the GNU Library General Public
    License as published by the Free Software Foundation; either
    version 2 of the License, or (at your option) any later version.

    This library is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
    Library General Public License for more details.

    You should have received a copy of the GNU Library General Public
    License along with this library; if not, write to the Free
    Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
*/

#include "..\..\include\cssysdef.h"

void csAutoThreadedLoad::Run(csAutoLoader* aloader)
{
csRef<iVFS> vfs;
//TODO:initialize vfs.

csAutoLoaderJob& job;

//aloader->current->SetComplete(0);
//aloader->current->SetTotal(aloader->jobs.GetSize());

csArray<csAutoLoaderJob>::Iterator it;
it = aloader->jobs->GetIterator();

while(it.HasNext())
{
jobs = it.Next();
csRef<iFile> file = vfs->Open(job.fname, VFS_FILE_READ);
job.data = file->GetAllData();
//aloader->current->Step();
job.done = true;
}
//aloader->current->Finished();
aloader->jobs.Empty();

aloading->loading = false;
}

csAutoLoader::csAutoLoader()
{
loading  = false;
}
void csAutoLoader::AddJob(const char* name, const char* fname)
{
csAutoLoaderJob job;
job.fname = new char[strlen(fname)];
job.name = new char[strlen(name)];
strcpy(job.fname,fname);
strcpy(job.name,name);

jobs.Push(job);
}
void csAutoLoader::RemoveJob(const char* name)
{
int i;
for(i = 0;i<jobs.GetSize();i++)
{
if(strcmp(jobs.GetAt(i).name, name)==0)
{
csAutoLoaderJob job = jobs.GetAt(i);
delete [] job.fname;
delete [] job.name;
delete [] job.type;

jobs.DeleteIndex(i);
}
}
}
void csAutoLoader::StartLoad(/*iAutoLoaderProgress& progress = internal*/)
{
//current = progress;
loading = true;
csRef<csThread> thread = csThread::Create(new csAutoThreadedLoad);
thread->Start();
}
bool csAutoLoader::Finish()
{
if(!loading)
{
csRef<iLoader> loader;
//TODO: initialize loader

csArray<csAutoLoaderJob>::Iterator it;
it = jobs->GetIterator();

int i = 0;

while(it.HasNext())
{
job = it.Next();
if(job.done)
{
if(loader->Load(&job.data))
jobs->DeleteIndex(i);
}
i++;
}
return true;
}
else return false;
}

In CS\include\imap\csautoloader.h :

Code:

/*
    Copyright (C) 2006 by Joshua Warner

    This library is free software; you can redistribute it and/or
    modify it under the terms of the GNU Library General Public
    License as published by the Free Software Foundation; either
    version 2 of the License, or (at your option) any later version.

    This library is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
    Library General Public License for more details.

    You should have received a copy of the GNU Library General Public
    License along with this library; if not, write to the Free
    Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
*/

#ifndef __CS_AUTOLOADER_H__
#define __CS_AUTOLOADER_H__

//class csAutoLoaderProgress;
class csAutoLoader;
//class iAutoLoaderProgress;
class iAutoLoader;

/*class csAutoLoaderProgress : public iAutoLoaderProgress
{
public:
csAutoLoaderProgress(void){SetTotal(100);SetComplete(0);abort = false;finished = false;}
~csAutoLoaderProgress(void){return;}

virtual void Abort(){abort = true;}
virtual void Finalize(){finished = true;}
virtual void Finished(){finished = true;complete = 0;}
virtual bool IsFinished(){return finished;}
virtual bool IsAborted(){return abort;}
virtual float GetCurrent(){return ((float)complete/(float)total);}
virtual int GetTotal(){return total;}
virtual int GetComplete(){return complete;}
virtual void SetComplete(int num){complete = num;}
virtual void SetTotal(int num){total = num;}
virtual void Step(int num = 1){if(total>=complete)complete+=num;else Finished();}

private:
    int total;
int complete;
bool finished;
bool abort;
};*/

static struct csAutoLoaderJob
{
const char* name;
const char* fname;
bool done;
csRef<iDataBuffer> data;
};

static class csAutoThreadedLoad : public csRunnable
{
public:
virtual void Run(csAutoLoader* aloader);
};

class csAutoLoader : public iAutoLoader
{
public:
csAutoLoader();
~csAutoLoader();

struct Component : public iComponent
{
SCF_DECLARE_EMBEDDED_IBASE (MyPlugin);
virtual bool Initialize (iObjectRegistry* r)
{ return scfParent->Initialize (r); }
} scfiComponent;

virtual void AddJob(const char* name, const char* fname);
virtual void RemoveJob(const char* name);
virtual void StartLoad(/*iAutoLoaderProgress& progress = internal*/);

virtual void Initialize(iObjectRegistry* r){loader = CS_QUERY_REGISTRY(r, loader);}

virtual bool Finish();

//add custom map loading funtions.

private:
csArray<csAutoLoaderJob> jobs;
//csRef<iAutoLoaderProgress> internal;
//csRef<iAutoLoaderProgress> current;
bool loading;

csRef<iLoader> loader;

friend class csAutoThreadedLoad;
};

#endif // __CS_AUTOLOADER_H__

Please note that while reading this code, you will probably see many errors in it.  I ask only that someone tell me what I am doing wrong with interfacing it with CS.
What I am getting now is errors concerning include statements.  Also, it would be helpful if someone could give me some info on the use of the CS plugin define statements, as the documentation seems a little out of date on this issue.  (all the plugins that I have looked at the code for use a different method than what is described in the documentation.)

Thanks for voting on the pole - those that did.  FYI, the progress handeling code is commented out for debugging purposes.  It will be enabled once I get everything else working.

[jorrit]

Can you give us the exact errors you are getting? That would be very helpful.

Greetings,

[allpowerful32]

There have simply been a whole host of #include errors, the most recent, and the one that I am currently stuck on is that, for some reason I am getting a "cannot open include file" error in csplatform.h on the "#include <csconfig.h>" line.  Note that I havent modified any CS files : only added two.  Jorrit: i'd like an official concensus on this:  what progress metering technique should I use (see the pole)?  Maybe support several for various coding styles?

[jorrit]

iProgressMeter is the one you should use as that is the official one. Everyone can make implementations if iProgressMeter to do more interesting versions of that (see cslight application for an example).

Greetings,

[rvl_]

I'm using CS as mozilla plugin & i'm found that the simplest way to make loading in separate thread is to run whole CS in that thread. No errors, no problems - at least on Win32 platform.

[jorrit]

I'm using CS as mozilla plugin & i'm found that the simplest way to make loading in separate thread is to run whole CS in that thread. No errors, no problems - at least on Win32 platform.

Yes, but that's not what we want.  The idea is to have loading in a separate thread from the rendering thread.

Greetings,