hadoop mailing list atom feed

building the eclipse plugin

brien colwell — Sun, 28 Jun 2009 19:45:48 -0700

hi all -- Just wondering how to build the eclipse plugin. "ant binary" does not seem to catch it. I would like to experiment with a few changes. thanks!
Brien

Re: hadoop jobs take long time to setup

Mikhail Bautin — Sun, 28 Jun 2009 15:32:40 -0700

Marcus, The code that needs to patched is in the tasktracker, because the
tasktracker is what starts the child JVM that runs user code. Thanks,
Mikhail On Sun, Jun 28, 2009 at 6:14 PM, Marcus Herou <marcus.herou@tailsweep.com>wrote: > Hi.
>
> Just to be clear. It is the jobtracker that needs the patched code right ?
> Or is it the tasktrackers ?
>
> Kindly
>
> //Marcus
>
> On Mon, Jun 29, 2009 at 12:08 AM, Mikhail Bautin <mbautin@gmail.com>
> wrote:
>
> > Marcus,
> >
> > We currently use 0.20.0 but this patch just inserts 8 lines of code into
> > TaskRunner.java, which could certainly be done with 0.18.3.
> >
> > Yes, this patch just appends additional jars to the child JVM classpath.
> >
> > I've never really used tmpjars myself, but if it involves uploading
> > multiple
> > jar files into HDFS every time a job is started, I see how it can be
> really
> > slow. On our ~80-job workflow this would have really slowed things down.
> >
> > Thanks,
> > Mikhail
> >
> > On Sun, Jun 28, 2009 at 5:40 PM, Marcus Herou <
> marcus.herou@tailsweep.com
> > >wrote:
> >
> > > Makes sense... I will try both rsync and NFS but I think rsync will
> beat
> > > NFS
> > > since NFS can be slow as hell sometimes but what the heck we already
> have
> > > our maven2 repo on NFS so why not :)
> > >
> > > Are you saying that this patch make the client able to configure which
> > > "extra" local jar files to add as classpath when firing up the
> > > TaskTrackerChild ?
> > >
> > > To be explicit: Do you confirm that using tmpjars like I do is a
> costful
> > > slow operation ?
> > >
> > > To what branch to you apply the patch (we use 0.18.3) ?
> > >
> > > Cheers
> > >
> > > //Marcus
> > >
> > >
> > > On Sun, Jun 28, 2009 at 11:26 PM, Mikhail Bautin <mbautin@gmail.com>
> > > wrote:
> > >
> > > > This is the way we deal with this problem, too. We put our jar files
> on
> > > > NFS, and the attached patch makes possible to add those jar files to
> > the
> > > > tasktracker classpath through a configuration property.
> > > >
> > > > Thanks,
> > > > Mikhail
> > > >
> > > > On Sun, Jun 28, 2009 at 5:21 PM, Stuart White <
> stuart.white1@gmail.com
> > > >wrote:
> > > >
> > > >> Although I've never done it, I believe you could manually copy your
> > jar
> > > >> files out to your cluster somewhere in hadoop's classpath, and that
> > > would
> > > >> remove the need for you to copy them to your cluster at the start
of
> > > each
> > > >> job.
> > > >>
> > > >> On Sun, Jun 28, 2009 at 4:08 PM, Marcus Herou <
> > > marcus.herou@tailsweep.com
> > > >> >wrote:
> > > >>
> > > >> > Hi.
> > > >> >
> > > >> > Running without a jobtracker makes the job start almost instantly.
> > > >> > I think it is due to something with the classloader. I use a
huge
> > > amount
> > > >> of
> > > >> > jarfiles jobConf.set("tmpjars", "jar1.jar,jar2.jar")... which
need
> > to
> > > be
> > > >> > loaded every time I guess.
> > > >> >
> > > >> > By issuing conf.setNumTasksToExecutePerJvm(-1); will the
> TaskTracker
> > > >> child
> > > >> > live forever then ?
> > > >> >
> > > >> > Cheers
> > > >> >
> > > >> > //Marcus
> > > >> >
> > > >> > On Sun, Jun 28, 2009 at 9:54 PM, tim robertson <
> > > >> timrobertson100@gmail.com
> > > >> > >wrote:
> > > >> >
> > > >> > > How long does it take to start the code locally in a single
> > thread?
> > > >> > >
> > > >> > > Can you reuse the JVM so it only starts once per node per
job?
> > > >> > > conf.setNumTasksToExecutePerJvm(-1)
> > > >> > >
> > > >> > > Cheers,
> > > >> > > Tim
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > On Sun, Jun 28, 2009 at 9:43 PM, Marcus Herou<
> > > >> marcus.herou@tailsweep.com
> > > >> > >
> > > >> > > wrote:
> > > >> > > > Hi.
> > > >> > > >
> > > >> > > > Wonder how one should improve the startup times of
a hadoop
> job.
> > > >> Some
> > > >> > of
> > > >> > > my
> > > >> > > > jobs which have a lot of dependencies in terms of many
jar
> files
> > > >> take a
> > > >> > > long
> > > >> > > > time to start in hadoop up to 2 minutes some times.
> > > >> > > > The data input amounts in these cases are neglible
so it seems
> > > that
> > > >> > > Hadoop
> > > >> > > > have a really high setup cost, which I can live with
but this
> > > seems
> > > >> to
> > > >> > > much.
> > > >> > > >
> > > >> > > > Let's say a job takes 10 minutes to complete then it
is bad if
> > it
> > > >> takes
> > > >> > 2
> > > >> > > > mins to set it up... 20-30 sec max would be a lot more
> > reasonable.
> > > >> > > >
> > > >> > > > Hints ?
> > > >> > > >
> > > >> > > > //Marcus
> > > >> > > >
> > > >> > > >
> > > >> > > > --
> > > >> > > > Marcus Herou CTO and co-founder Tailsweep AB
> > > >> > > > +46702561312
> > > >> > > > marcus.herou@tailsweep.com
> > > >> > > > http://www.tailsweep.com/
> > > >> > > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > Marcus Herou CTO and co-founder Tailsweep AB
> > > >> > +46702561312
> > > >> > marcus.herou@tailsweep.com
> > > >> > http://www.tailsweep.com/
> > > >> >
> > > >>
> > > >
> > > >
> > >
> > >
> >
>
>
>
> --
> Marcus Herou CTO and co-founder Tailsweep AB
> +46702561312
> marcus.herou@tailsweep.com
> http://www.tailsweep.com/
>

Re: hadoop jobs take long time to setup

Marcus Herou — Sun, 28 Jun 2009 15:14:33 -0700

Hi. Just to be clear. It is the jobtracker that needs the patched code right ?
Or is it the tasktrackers ? Kindly //Marcus On Mon, Jun 29, 2009 at 12:08 AM, Mikhail Bautin <mbautin@gmail.com> wrote: > Marcus,
>
> We currently use 0.20.0 but this patch just inserts 8 lines of code into
> TaskRunner.java, which could certainly be done with 0.18.3.
>
> Yes, this patch just appends additional jars to the child JVM classpath.
>
> I've never really used tmpjars myself, but if it involves uploading
> multiple
> jar files into HDFS every time a job is started, I see how it can be really
> slow. On our ~80-job workflow this would have really slowed things down.
>
> Thanks,
> Mikhail
>
> On Sun, Jun 28, 2009 at 5:40 PM, Marcus Herou <marcus.herou@tailsweep.com
> >wrote:
>
> > Makes sense... I will try both rsync and NFS but I think rsync will beat
> > NFS
> > since NFS can be slow as hell sometimes but what the heck we already have
> > our maven2 repo on NFS so why not :)
> >
> > Are you saying that this patch make the client able to configure which
> > "extra" local jar files to add as classpath when firing up the
> > TaskTrackerChild ?
> >
> > To be explicit: Do you confirm that using tmpjars like I do is a costful
> > slow operation ?
> >
> > To what branch to you apply the patch (we use 0.18.3) ?
> >
> > Cheers
> >
> > //Marcus
> >
> >
> > On Sun, Jun 28, 2009 at 11:26 PM, Mikhail Bautin <mbautin@gmail.com>
> > wrote:
> >
> > > This is the way we deal with this problem, too. We put our jar files on
> > > NFS, and the attached patch makes possible to add those jar files to
> the
> > > tasktracker classpath through a configuration property.
> > >
> > > Thanks,
> > > Mikhail
> > >
> > > On Sun, Jun 28, 2009 at 5:21 PM, Stuart White <stuart.white1@gmail.com
> > >wrote:
> > >
> > >> Although I've never done it, I believe you could manually copy your
> jar
> > >> files out to your cluster somewhere in hadoop's classpath, and that
> > would
> > >> remove the need for you to copy them to your cluster at the start of
> > each
> > >> job.
> > >>
> > >> On Sun, Jun 28, 2009 at 4:08 PM, Marcus Herou <
> > marcus.herou@tailsweep.com
> > >> >wrote:
> > >>
> > >> > Hi.
> > >> >
> > >> > Running without a jobtracker makes the job start almost instantly.
> > >> > I think it is due to something with the classloader. I use a huge
> > amount
> > >> of
> > >> > jarfiles jobConf.set("tmpjars", "jar1.jar,jar2.jar")... which need
> to
> > be
> > >> > loaded every time I guess.
> > >> >
> > >> > By issuing conf.setNumTasksToExecutePerJvm(-1); will the TaskTracker
> > >> child
> > >> > live forever then ?
> > >> >
> > >> > Cheers
> > >> >
> > >> > //Marcus
> > >> >
> > >> > On Sun, Jun 28, 2009 at 9:54 PM, tim robertson <
> > >> timrobertson100@gmail.com
> > >> > >wrote:
> > >> >
> > >> > > How long does it take to start the code locally in a single
> thread?
> > >> > >
> > >> > > Can you reuse the JVM so it only starts once per node per job?
> > >> > > conf.setNumTasksToExecutePerJvm(-1)
> > >> > >
> > >> > > Cheers,
> > >> > > Tim
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Sun, Jun 28, 2009 at 9:43 PM, Marcus Herou<
> > >> marcus.herou@tailsweep.com
> > >> > >
> > >> > > wrote:
> > >> > > > Hi.
> > >> > > >
> > >> > > > Wonder how one should improve the startup times of a hadoop
job.
> > >> Some
> > >> > of
> > >> > > my
> > >> > > > jobs which have a lot of dependencies in terms of many jar
files
> > >> take a
> > >> > > long
> > >> > > > time to start in hadoop up to 2 minutes some times.
> > >> > > > The data input amounts in these cases are neglible so it
seems
> > that
> > >> > > Hadoop
> > >> > > > have a really high setup cost, which I can live with but
this
> > seems
> > >> to
> > >> > > much.
> > >> > > >
> > >> > > > Let's say a job takes 10 minutes to complete then it is
bad if
> it
> > >> takes
> > >> > 2
> > >> > > > mins to set it up... 20-30 sec max would be a lot more
> reasonable.
> > >> > > >
> > >> > > > Hints ?
> > >> > > >
> > >> > > > //Marcus
> > >> > > >
> > >> > > >
> > >> > > > --
> > >> > > > Marcus Herou CTO and co-founder Tailsweep AB
> > >> > > > +46702561312
> > >> > > > marcus.herou@tailsweep.com
> > >> > > > http://www.tailsweep.com/
> > >> > > >
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Marcus Herou CTO and co-founder Tailsweep AB
> > >> > +46702561312
> > >> > marcus.herou@tailsweep.com
> > >> > http://www.tailsweep.com/
> > >> >
> > >>
> > >
> > >
> >
> >
> -- Marcus Herou CTO and co-founder Tailsweep AB
+46702561312
marcus.herou@tailsweep.com
http://www.tailsweep.com/

Re: hadoop jobs take long time to setup

Mikhail Bautin — Sun, 28 Jun 2009 15:08:08 -0700

Marcus, We currently use 0.20.0 but this patch just inserts 8 lines of code into
TaskRunner.java, which could certainly be done with 0.18.3. Yes, this patch just appends additional jars to the child JVM classpath. I've never really used tmpjars myself, but if it involves uploading multiple
jar files into HDFS every time a job is started, I see how it can be really
slow. On our ~80-job workflow this would have really slowed things down. Thanks,
Mikhail On Sun, Jun 28, 2009 at 5:40 PM, Marcus Herou <marcus.herou@tailsweep.com>wrote: > Makes sense... I will try both rsync and NFS but I think rsync will beat
> NFS
> since NFS can be slow as hell sometimes but what the heck we already have
> our maven2 repo on NFS so why not :)
>
> Are you saying that this patch make the client able to configure which
> "extra" local jar files to add as classpath when firing up the
> TaskTrackerChild ?
>
> To be explicit: Do you confirm that using tmpjars like I do is a costful
> slow operation ?
>
> To what branch to you apply the patch (we use 0.18.3) ?
>
> Cheers
>
> //Marcus
>
>
> On Sun, Jun 28, 2009 at 11:26 PM, Mikhail Bautin <mbautin@gmail.com>
> wrote:
>
> > This is the way we deal with this problem, too. We put our jar files on
> > NFS, and the attached patch makes possible to add those jar files to the
> > tasktracker classpath through a configuration property.
> >
> > Thanks,
> > Mikhail
> >
> > On Sun, Jun 28, 2009 at 5:21 PM, Stuart White <stuart.white1@gmail.com
> >wrote:
> >
> >> Although I've never done it, I believe you could manually copy your jar
> >> files out to your cluster somewhere in hadoop's classpath, and that
> would
> >> remove the need for you to copy them to your cluster at the start of
> each
> >> job.
> >>
> >> On Sun, Jun 28, 2009 at 4:08 PM, Marcus Herou <
> marcus.herou@tailsweep.com
> >> >wrote:
> >>
> >> > Hi.
> >> >
> >> > Running without a jobtracker makes the job start almost instantly.
> >> > I think it is due to something with the classloader. I use a huge
> amount
> >> of
> >> > jarfiles jobConf.set("tmpjars", "jar1.jar,jar2.jar")... which need to
> be
> >> > loaded every time I guess.
> >> >
> >> > By issuing conf.setNumTasksToExecutePerJvm(-1); will the TaskTracker
> >> child
> >> > live forever then ?
> >> >
> >> > Cheers
> >> >
> >> > //Marcus
> >> >
> >> > On Sun, Jun 28, 2009 at 9:54 PM, tim robertson <
> >> timrobertson100@gmail.com
> >> > >wrote:
> >> >
> >> > > How long does it take to start the code locally in a single thread?
> >> > >
> >> > > Can you reuse the JVM so it only starts once per node per job?
> >> > > conf.setNumTasksToExecutePerJvm(-1)
> >> > >
> >> > > Cheers,
> >> > > Tim
> >> > >
> >> > >
> >> > >
> >> > > On Sun, Jun 28, 2009 at 9:43 PM, Marcus Herou<
> >> marcus.herou@tailsweep.com
> >> > >
> >> > > wrote:
> >> > > > Hi.
> >> > > >
> >> > > > Wonder how one should improve the startup times of a hadoop job.
> >> Some
> >> > of
> >> > > my
> >> > > > jobs which have a lot of dependencies in terms of many jar files
> >> take a
> >> > > long
> >> > > > time to start in hadoop up to 2 minutes some times.
> >> > > > The data input amounts in these cases are neglible so it seems
> that
> >> > > Hadoop
> >> > > > have a really high setup cost, which I can live with but this
> seems
> >> to
> >> > > much.
> >> > > >
> >> > > > Let's say a job takes 10 minutes to complete then it is bad if
it
> >> takes
> >> > 2
> >> > > > mins to set it up... 20-30 sec max would be a lot more reasonable.
> >> > > >
> >> > > > Hints ?
> >> > > >
> >> > > > //Marcus
> >> > > >
> >> > > >
> >> > > > --
> >> > > > Marcus Herou CTO and co-founder Tailsweep AB
> >> > > > +46702561312
> >> > > > marcus.herou@tailsweep.com
> >> > > > http://www.tailsweep.com/
> >> > > >
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > Marcus Herou CTO and co-founder Tailsweep AB
> >> > +46702561312
> >> > marcus.herou@tailsweep.com
> >> > http://www.tailsweep.com/
> >> >
> >>
> >
> >
>
>

Re: FSDataInputStream.read(byte[]) only reads to a block boundary?

"M. C. Srivas" — Sun, 28 Jun 2009 15:07:46 -0700

On Sun, Jun 28, 2009 at 3:01 PM, Matei Zaharia <matei@cloudera.com> wrote: > This kind of partial read is often used by the OS to return to your
> application as soon as possible if trying to read more data would block, in
> case you can begin computing on the partial data. In some applications,
> it's
> not useful, but when you can begin computing on partial data, it allows the
> OS to overlap IO with your computation, improving throughput. I think
> FSDataInputStream returns at the block boundary for the same reason.
> It is very unusual, nay, unexpected to the point of bizarre, for the OS to
do so on a regular file. Typically only seen on network fds. >
> On Sun, Jun 28, 2009 at 11:16 AM, Raghu Angadi <rangadi@yahoo-inc.com
> >wrote:
>
> >
> > This seems to be the case. I don't think there is any specific reason not
> > to read across the block boundary...
> >
> > Even if HDFS does read across the blocks, it is still not a good idea to
> > ignore the JavaDoc for read(). If you want all the bytes read, then you
> > should have a while loop or one of the readFully() variants. For e.g. if
> you
> > later change your code by wrapping a BufferedInputStream around 'in', you
> > would still get partial reads even if HDFS reads all the data.
> >
> > Raghu.
> >
> >
> > forbbs forbbs wrote:
> >
> >> The hadoop version is 0.19.0.
> >> My file is larger than 64MB, and the block size is 64MB.
> >>
> >> The output of the code below is '10'. May I read across the block
> >> boundary? Or I should use 'while (left..){}' style code?
> >>
> >> public static void main(String[] args) throws IOException
> >> {
> >> Configuration conf = new Configuration();
> >> FileSystem fs = FileSystem.get(conf);
> >> FSDataInputStream fin = fs.open(new Path(args[0]));
> >>
> >> fin.seek(64*1024*1024 - 10);
> >> byte[] buffer = new byte[32*1024];
> >> int len = fin.read(buffer);
> >> //int len = fin.read(buffer, 0, 128);
> >> System.out.println(len);
> >>
> >> fin.close();
> >> }
> >>
> >
> >
>

Re: FSDataInputStream.read(byte[]) only reads to a block boundary?

Matei Zaharia — Sun, 28 Jun 2009 15:01:37 -0700

This kind of partial read is often used by the OS to return to your
application as soon as possible if trying to read more data would block, in
case you can begin computing on the partial data. In some applications, it's
not useful, but when you can begin computing on partial data, it allows the
OS to overlap IO with your computation, improving throughput. I think
FSDataInputStream returns at the block boundary for the same reason. On Sun, Jun 28, 2009 at 11:16 AM, Raghu Angadi <rangadi@yahoo-inc.com>wrote: >
> This seems to be the case. I don't think there is any specific reason not
> to read across the block boundary...
>
> Even if HDFS does read across the blocks, it is still not a good idea to
> ignore the JavaDoc for read(). If you want all the bytes read, then you
> should have a while loop or one of the readFully() variants. For e.g. if you
> later change your code by wrapping a BufferedInputStream around 'in', you
> would still get partial reads even if HDFS reads all the data.
>
> Raghu.
>
>
> forbbs forbbs wrote:
>
>> The hadoop version is 0.19.0.
>> My file is larger than 64MB, and the block size is 64MB.
>>
>> The output of the code below is '10'. May I read across the block
>> boundary? Or I should use 'while (left..){}' style code?
>>
>> public static void main(String[] args) throws IOException
>> {
>> Configuration conf = new Configuration();
>> FileSystem fs = FileSystem.get(conf);
>> FSDataInputStream fin = fs.open(new Path(args[0]));
>>
>> fin.seek(64*1024*1024 - 10);
>> byte[] buffer = new byte[32*1024];
>> int len = fin.read(buffer);
>> //int len = fin.read(buffer, 0, 128);
>> System.out.println(len);
>>
>> fin.close();
>> }
>>
>
>

Re: hadoop jobs take long time to setup

Marcus Herou — Sun, 28 Jun 2009 14:40:01 -0700

Makes sense... I will try both rsync and NFS but I think rsync will beat NFS
since NFS can be slow as hell sometimes but what the heck we already have
our maven2 repo on NFS so why not :) Are you saying that this patch make the client able to configure which
"extra" local jar files to add as classpath when firing up the
TaskTrackerChild ? To be explicit: Do you confirm that using tmpjars like I do is a costful
slow operation ? To what branch to you apply the patch (we use 0.18.3) ? Cheers //Marcus On Sun, Jun 28, 2009 at 11:26 PM, Mikhail Bautin <mbautin@gmail.com> wrote: > This is the way we deal with this problem, too. We put our jar files on
> NFS, and the attached patch makes possible to add those jar files to the
> tasktracker classpath through a configuration property.
>
> Thanks,
> Mikhail
>
> On Sun, Jun 28, 2009 at 5:21 PM, Stuart White <stuart.white1@gmail.com>wrote:
>
>> Although I've never done it, I believe you could manually copy your jar
>> files out to your cluster somewhere in hadoop's classpath, and that would
>> remove the need for you to copy them to your cluster at the start of each
>> job.
>>
>> On Sun, Jun 28, 2009 at 4:08 PM, Marcus Herou <marcus.herou@tailsweep.com
>> >wrote:
>>
>> > Hi.
>> >
>> > Running without a jobtracker makes the job start almost instantly.
>> > I think it is due to something with the classloader. I use a huge amount
>> of
>> > jarfiles jobConf.set("tmpjars", "jar1.jar,jar2.jar")... which need to be
>> > loaded every time I guess.
>> >
>> > By issuing conf.setNumTasksToExecutePerJvm(-1); will the TaskTracker
>> child
>> > live forever then ?
>> >
>> > Cheers
>> >
>> > //Marcus
>> >
>> > On Sun, Jun 28, 2009 at 9:54 PM, tim robertson <
>> timrobertson100@gmail.com
>> > >wrote:
>> >
>> > > How long does it take to start the code locally in a single thread?
>> > >
>> > > Can you reuse the JVM so it only starts once per node per job?
>> > > conf.setNumTasksToExecutePerJvm(-1)
>> > >
>> > > Cheers,
>> > > Tim
>> > >
>> > >
>> > >
>> > > On Sun, Jun 28, 2009 at 9:43 PM, Marcus Herou<
>> marcus.herou@tailsweep.com
>> > >
>> > > wrote:
>> > > > Hi.
>> > > >
>> > > > Wonder how one should improve the startup times of a hadoop job.
>> Some
>> > of
>> > > my
>> > > > jobs which have a lot of dependencies in terms of many jar files
>> take a
>> > > long
>> > > > time to start in hadoop up to 2 minutes some times.
>> > > > The data input amounts in these cases are neglible so it seems that
>> > > Hadoop
>> > > > have a really high setup cost, which I can live with but this seems
>> to
>> > > much.
>> > > >
>> > > > Let's say a job takes 10 minutes to complete then it is bad if it
>> takes
>> > 2
>> > > > mins to set it up... 20-30 sec max would be a lot more reasonable.
>> > > >
>> > > > Hints ?
>> > > >
>> > > > //Marcus
>> > > >
>> > > >
>> > > > --
>> > > > Marcus Herou CTO and co-founder Tailsweep AB
>> > > > +46702561312
>> > > > marcus.herou@tailsweep.com
>> > > > http://www.tailsweep.com/
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Marcus Herou CTO and co-founder Tailsweep AB
>> > +46702561312
>> > marcus.herou@tailsweep.com
>> > http://www.tailsweep.com/
>> >
>>
>
> -- Marcus Herou CTO and co-founder Tailsweep AB
+46702561312
marcus.herou@tailsweep.com
http://www.tailsweep.com/

Re: hadoop jobs take long time to setup

Mikhail Bautin — Sun, 28 Jun 2009 14:26:02 -0700

An error occured while fetching this message, sorry !

Re: hadoop jobs take long time to setup

Stuart White — Sun, 28 Jun 2009 14:21:35 -0700

Although I've never done it, I believe you could manually copy your jar
files out to your cluster somewhere in hadoop's classpath, and that would
remove the need for you to copy them to your cluster at the start of each
job. On Sun, Jun 28, 2009 at 4:08 PM, Marcus Herou <marcus.herou@tailsweep.com>wrote: > Hi.
>
> Running without a jobtracker makes the job start almost instantly.
> I think it is due to something with the classloader. I use a huge amount of
> jarfiles jobConf.set("tmpjars", "jar1.jar,jar2.jar")... which need to be
> loaded every time I guess.
>
> By issuing conf.setNumTasksToExecutePerJvm(-1); will the TaskTracker child
> live forever then ?
>
> Cheers
>
> //Marcus
>
> On Sun, Jun 28, 2009 at 9:54 PM, tim robertson <timrobertson100@gmail.com
> >wrote:
>
> > How long does it take to start the code locally in a single thread?
> >
> > Can you reuse the JVM so it only starts once per node per job?
> > conf.setNumTasksToExecutePerJvm(-1)
> >
> > Cheers,
> > Tim
> >
> >
> >
> > On Sun, Jun 28, 2009 at 9:43 PM, Marcus Herou<marcus.herou@tailsweep.com
> >
> > wrote:
> > > Hi.
> > >
> > > Wonder how one should improve the startup times of a hadoop job. Some
> of
> > my
> > > jobs which have a lot of dependencies in terms of many jar files take a
> > long
> > > time to start in hadoop up to 2 minutes some times.
> > > The data input amounts in these cases are neglible so it seems that
> > Hadoop
> > > have a really high setup cost, which I can live with but this seems to
> > much.
> > >
> > > Let's say a job takes 10 minutes to complete then it is bad if it takes
> 2
> > > mins to set it up... 20-30 sec max would be a lot more reasonable.
> > >
> > > Hints ?
> > >
> > > //Marcus
> > >
> > >
> > > --
> > > Marcus Herou CTO and co-founder Tailsweep AB
> > > +46702561312
> > > marcus.herou@tailsweep.com
> > > http://www.tailsweep.com/
> > >
> >
>
>
>
> --
> Marcus Herou CTO and co-founder Tailsweep AB
> +46702561312
> marcus.herou@tailsweep.com
> http://www.tailsweep.com/
>

Re: hadoop jobs take long time to setup

Marcus Herou — Sun, 28 Jun 2009 14:08:15 -0700

Hi. Running without a jobtracker makes the job start almost instantly.
I think it is due to something with the classloader. I use a huge amount of
jarfiles jobConf.set("tmpjars", "jar1.jar,jar2.jar")... which need to be
loaded every time I guess. By issuing conf.setNumTasksToExecutePerJvm(-1); will the TaskTracker child
live forever then ? Cheers //Marcus On Sun, Jun 28, 2009 at 9:54 PM, tim robertson <timrobertson100@gmail.com>wrote: > How long does it take to start the code locally in a single thread?
>
> Can you reuse the JVM so it only starts once per node per job?
> conf.setNumTasksToExecutePerJvm(-1)
>
> Cheers,
> Tim
>
>
>
> On Sun, Jun 28, 2009 at 9:43 PM, Marcus Herou<marcus.herou@tailsweep.com>
> wrote:
> > Hi.
> >
> > Wonder how one should improve the startup times of a hadoop job. Some of
> my
> > jobs which have a lot of dependencies in terms of many jar files take a
> long
> > time to start in hadoop up to 2 minutes some times.
> > The data input amounts in these cases are neglible so it seems that
> Hadoop
> > have a really high setup cost, which I can live with but this seems to
> much.
> >
> > Let's say a job takes 10 minutes to complete then it is bad if it takes 2
> > mins to set it up... 20-30 sec max would be a lot more reasonable.
> >
> > Hints ?
> >
> > //Marcus
> >
> >
> > --
> > Marcus Herou CTO and co-founder Tailsweep AB
> > +46702561312
> > marcus.herou@tailsweep.com
> > http://www.tailsweep.com/
> >
> -- Marcus Herou CTO and co-founder Tailsweep AB
+46702561312
marcus.herou@tailsweep.com
http://www.tailsweep.com/

Re: Scaling out/up or a mix

Marcus Herou — Sun, 28 Jun 2009 13:49:11 -0700

Hi. The crawlers are _very_ threaded but no we use our own threading framework
since it was not available at the time on hadoop-core. Crawlers normally just wait a lot on clients inducing very little CPU but
consumes some memory due to the parallellism. //Marcus On Sat, Jun 27, 2009 at 6:10 PM, jason hadoop <jason.hadoop@gmail.com>wrote: > How about multi-threaded mappers?
> Multi-Threaded mappers are ideal for map tasks that are non locally io
> bound
> with many distinct endpoints.
> You can also control the thread count on a per job basis.
>
> On Sat, Jun 27, 2009 at 8:26 AM, Marcus Herou <marcus.herou@tailsweep.com
> >wrote:
>
> > The argument currently against increasing num-mappers is that the
> machines
> > will get into oom and since a lot of the jobs are crawlers I need more
> > ip-numbers so I don't get banned :)
> >
> > Thing is that we currently have solr on the very same machines and
> > data-nodes as well so I can only give the MR nodes about 1G memory since
> I
> > need SOLR to have 4G...
> >
> > Now I see that I should get some obvious and juste critique about the
> > layout
> > of this arch but I'm a little limited in budget and so is then the arch
> :)
> >
> > However is it wise to have the MR tasks on the same nodes as the
> data-nodes
> > or should I split the arch ? I mean the data-nodes perhaps need more
> > disk-IO
> > and the MR more memory and CPU ?
> >
> > Trying to find a sweetspot hardware spec of those two roles.
> >
> > //Marcus
> >
> >
> >
> > On Sat, Jun 27, 2009 at 4:24 AM, Brian Bockelman <bbockelm@cse.unl.edu
> > >wrote:
> >
> > > Hey Marcus,
> > >
> > > Are you recording the data rates coming out of HDFS? Since you have
> such
> > a
> > > low CPU utilizations, I'd look at boxes utterly packed with big hard
> > drives
> > > (also, why are you using RAID1 for Hadoop??).
> > >
> > > You can get 1U boxes with 4 drive bays or 2U boxes with 12 drive bays.
> > > Based on the data rates you see, make the call.
> > >
> > > On the other hand, what's the argument against running 3x more mappers
> > per
> > > box? It seems that your boxes still have more overhead to use --
> there's
> > no
> > > I/O wait.
> > >
> > > Brian
> > >
> > >
> > > On Jun 26, 2009, at 4:43 PM, Marcus Herou wrote:
> > >
> > > Hi.
> > >>
> > >> We have a deployment of 10 hadoop servers and I now need more mapping
> > >> capability (no not just add more mappers per instance) since I have so
> > >> many
> > >> jobs running. Now I am wondering what I should aim on...
> > >> Memory, cpu or disk... How long is a rope perhaps you would say ?
> > >>
> > >> A typical server is currently using about 15-20% cpu today on a
> > quad-core
> > >> 2.4Ghz 8GB RAM machine with 2 RAID1 SATA 500GB disks.
> > >>
> > >> Some specs below.
> > >>
> > >>> mpstat 2 5
> > >>>
> > >> Linux 2.6.24-19-server (mapreduce2) 06/26/2009
> > >>
> > >> 11:36:13 PM CPU %user %nice %sys %iowait %irq %soft
> %steal
> > >> %idle intr/s
> > >> 11:36:15 PM all 22.82 0.00 3.24 1.37 0.62 2.49
> 0.00
> > >> 69.45 8572.50
> > >> 11:36:17 PM all 13.56 0.00 1.74 1.99 0.62 2.61
> 0.00
> > >> 79.48 8075.50
> > >> 11:36:19 PM all 14.32 0.00 2.24 1.12 1.12 2.24
> 0.00
> > >> 78.95 9219.00
> > >> 11:36:21 PM all 14.71 0.00 0.87 1.62 0.25 1.75
> 0.00
> > >> 80.80 8489.50
> > >> 11:36:23 PM all 12.69 0.00 0.87 1.24 0.50 0.75
> 0.00
> > >> 83.96 5495.00
> > >> Average: all 15.62 0.00 1.79 1.47 0.62 1.97
> 0.00
> > >> 78.53 7970.30
> > >>
> > >> What I am thinking is... Is it wiser to go for many of these cheap
> boxes
> > >> with 8GB of RAM or should I for instance focus on machines which can
> > give
> > >> more I|O throughput ?
> > >>
> > >> I know that these things are hard but perhaps someone have draw some
> > >> conclusions before the pragmatic way.
> > >>
> > >> Kindly
> > >>
> > >> //Marcus
> > >>
> > >>
> > >> --
> > >> Marcus Herou CTO and co-founder Tailsweep AB
> > >> +46702561312
> > >> marcus.herou@tailsweep.com
> > >> http://www.tailsweep.com/
> > >>
> > >
> > >
> >
> >
> > --
> > Marcus Herou CTO and co-founder Tailsweep AB
> > +46702561312
> > marcus.herou@tailsweep.com
> > http://www.tailsweep.com/
> >
>
>
>
> --
> Pro Hadoop, a book to guide you from beginner to hadoop mastery,
> http://www.amazon.com/dp/1430219424?tag=jewlerymall
> www.prohadoopbook.com a community for Hadoop Professionals
> -- Marcus Herou CTO and co-founder Tailsweep AB
+46702561312
marcus.herou@tailsweep.com
http://www.tailsweep.com/

Re: combine two map tasks

jason hadoop — Sun, 28 Jun 2009 13:06:55 -0700

The ChainMapper class introduced in Hadoop 19 will provide you with the
ability to have an arbitrary number of map tasks to run one after the other,
in the context of a single job.
The one issue to be aware of is that the chain of mappers only see the
output the previous map in the chain. There is a nice discussion of this in chapter 8 of Pro Hadoop, by Apress.com On Sun, Jun 28, 2009 at 5:04 AM, bharath vissapragada <
bharathvissapragada1990@gmail.com> wrote: > See this .. hope this answers your question .
>
> http://developer.yahoo.com/hadoop/tutorial/module4.html#tips
>
> On Sun, Jun 28, 2009 at 5:28 PM, bonito <bonito.perdo@gmail.com> wrote:
>
> >
> > Hello!
> > I am a new hadoop user and my question may sound naive..
> > However, I would like to ask if there is a way to combine the results of
> > two
> > mpa tasks that may "run" simultaneously.
> > I use the MultipleInput class and thus I have two different mappers.
> > I want the result/output of the one map (associated with one input file)
> to
> > be used in the process of the second map (associated with the second
> input
> > file).
> > I have thought of storing the map1 output in the hdfs and retrieving it
> > using the map2.
> > However, I have no clue whether this is possible. I mean...what about
> > time-executing issues? map2 has to wait until map1 is completed...
> >
> > The thought of executing them in a serial manner is not the one I really
> > want...
> >
> > Any suggestion would be appreciated.
> > Thank you in advance :)
> >
> > --
> > View this message in context:
> > http://www.nabble.com/combine-two-map-tasks-tp24240928p24240928.html
> > Sent from the Hadoop core-user mailing list archive at Nabble.com.
> >
> >
> -- Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals

Re: hadoop jobs take long time to setup

tim robertson — Sun, 28 Jun 2009 12:54:15 -0700

How long does it take to start the code locally in a single thread? Can you reuse the JVM so it only starts once per node per job?
conf.setNumTasksToExecutePerJvm(-1) Cheers,
Tim On Sun, Jun 28, 2009 at 9:43 PM, Marcus Herou<marcus.herou@tailsweep.com> wrote:
> Hi.
>
> Wonder how one should improve the startup times of a hadoop job. Some of my
> jobs which have a lot of dependencies in terms of many jar files take a long
> time to start in hadoop up to 2 minutes some times.
> The data input amounts in these cases are neglible so it seems that Hadoop
> have a really high setup cost, which I can live with but this seems to much.
>
> Let's say a job takes 10 minutes to complete then it is bad if it takes 2
> mins to set it up... 20-30 sec max would be a lot more reasonable.
>
> Hints ?
>
> //Marcus
>
>
> --
> Marcus Herou CTO and co-founder Tailsweep AB
> +46702561312
> marcus.herou@tailsweep.com
> http://www.tailsweep.com/
>

hadoop jobs take long time to setup

Marcus Herou — Sun, 28 Jun 2009 12:43:35 -0700

Hi. Wonder how one should improve the startup times of a hadoop job. Some of my
jobs which have a lot of dependencies in terms of many jar files take a long
time to start in hadoop up to 2 minutes some times.
The data input amounts in these cases are neglible so it seems that Hadoop
have a really high setup cost, which I can live with but this seems to much. Let's say a job takes 10 minutes to complete then it is bad if it takes 2
mins to set it up... 20-30 sec max would be a lot more reasonable. Hints ? //Marcus -- Marcus Herou CTO and co-founder Tailsweep AB
+46702561312
marcus.herou@tailsweep.com
http://www.tailsweep.com/

Re: FSDataInputStream.read(byte[]) only reads to a block boundary?

Raghu Angadi — Sun, 28 Jun 2009 11:16:05 -0700

 This seems to be the case. I don't think there is any specific reason not to read across the block boundary... Even if HDFS does read across the blocks, it is still not a good idea to ignore the JavaDoc for read(). If you want all the bytes read, then you should have a while loop or one of the readFully() variants. For e.g. if you later change your code by wrapping a BufferedInputStream around 'in', you would still get partial reads even if HDFS reads all the data. Raghu. forbbs forbbs wrote:
> The hadoop version is 0.19.0.
> My file is larger than 64MB, and the block size is 64MB.
> > The output of the code below is '10'. May I read across the block
> boundary? Or I should use 'while (left..){}' style code?
> > public static void main(String[] args) throws IOException
> {
> Configuration conf = new Configuration();
> FileSystem fs = FileSystem.get(conf);
> FSDataInputStream fin = fs.open(new Path(args[0]));
> > fin.seek(64*1024*1024 - 10);
> byte[] buffer = new byte[32*1024];
> int len = fin.read(buffer);
> //int len = fin.read(buffer, 0, 128);
> System.out.println(len);
> > fin.close();
> }

Re: combine two map tasks

bharath vissapragada — Sun, 28 Jun 2009 05:04:21 -0700

See this .. hope this answers your question . http://developer.yahoo.com/hadoop/tutorial/module4.html#tips On Sun, Jun 28, 2009 at 5:28 PM, bonito <bonito.perdo@gmail.com> wrote: >
> Hello!
> I am a new hadoop user and my question may sound naive..
> However, I would like to ask if there is a way to combine the results of
> two
> mpa tasks that may "run" simultaneously.
> I use the MultipleInput class and thus I have two different mappers.
> I want the result/output of the one map (associated with one input file) to
> be used in the process of the second map (associated with the second input
> file).
> I have thought of storing the map1 output in the hdfs and retrieving it
> using the map2.
> However, I have no clue whether this is possible. I mean...what about
> time-executing issues? map2 has to wait until map1 is completed...
>
> The thought of executing them in a serial manner is not the one I really
> want...
>
> Any suggestion would be appreciated.
> Thank you in advance :)
>
> --
> View this message in context:
> http://www.nabble.com/combine-two-map-tasks-tp24240928p24240928.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>

combine two map tasks

bonito — Sun, 28 Jun 2009 04:58:12 -0700

 Hello! I am a new hadoop user and my question may sound naive..
However, I would like to ask if there is a way to combine the results of two
mpa tasks that may "run" simultaneously. I use the MultipleInput class and thus I have two different mappers. I want the result/output of the one map (associated with one input file) to
be used in the process of the second map (associated with the second input
file). I have thought of storing the map1 output in the hdfs and retrieving it
using the map2.
However, I have no clue whether this is possible. I mean...what about
time-executing issues? map2 has to wait until map1 is completed... The thought of executing them in a serial manner is not the one I really
want... Any suggestion would be appreciated.
Thank you in advance :) -- View this message in context: http://www.nabble.com/combine-two-map-tasks-tp24240928p24240928.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

FSDataInputStream.read(byte[]) only reads to a block boundary?

forbbs forbbs — Sun, 28 Jun 2009 02:08:35 -0700

The hadoop version is 0.19.0.
My file is larger than 64MB, and the block size is 64MB. The output of the code below is '10'. May I read across the block
boundary? Or I should use 'while (left..){}' style code? public static void main(String[] args) throws IOException { Configuration conf = new Configuration(); FileSystem fs = FileSystem.get(conf); FSDataInputStream fin = fs.open(new Path(args[0])); fin.seek(64*1024*1024 - 10); byte[] buffer = new byte[32*1024]; int len = fin.read(buffer); //int len = fin.read(buffer, 0, 128); System.out.println(len); fin.close(); }

Re: hadoop lucene integration

zsongbo — Sat, 27 Jun 2009 11:01:47 -0700

katta ? On Fri, Jun 26, 2009 at 1:01 PM, Nick Cen <cenyongh@gmail.com> wrote: > there is sample index code under the contrib directory, maybe you can take
> a
> see.
>
> 2009/6/26 m.harig <m.harig@gmail.com>
>
> >
> > hi all
> >
> > I've work experience with lucene , but am new to hadoop , i
> > created a index by lucene , please any1 tell me how to use hadoop for my
> > lucene index for distributed file system , if possible can any1 send me
> an
> > example or the link in which i can use it for my index. Please .
> > --
> > View this message in context:
> > http://www.nabble.com/hadoop-lucene-integration-tp24214951p24214951.html
> > Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
> >
> >
>
>
> --
> http://daily.appspot.com/food/
>

Re: HDFS Random Access

Raghu Angadi — Sat, 27 Jun 2009 09:41:37 -0700

 Yes, FSDataInputStream allows random access. There are way to read x bytes at a position p:
1) in.seek(p); read(buf, 0, x);
2) in.(p, buf, 0, x);
These two have slightly different semantics. The second one is preferred and is easier for HDFS to optimize further. Random access should be pretty good with HDFS and it is increasingly getting more users and thus more importance. HBase is one of the users. Just yesterday I attached a benchmark and comparissions to random access on native filesystem to https://issues.apache.org/jira/browse/HDFS-236 . As of now, the overhead on average is about 2 ms over 9-10ms it takes for native read. There are a few fairly simple fixes possible to reduce this gap. I think getFileStatus() is the way to find the length, though there might have been a call added to FSDataInputStream recently. I am not sure. Raghu.
tsuraan wrote:
> All the documentation for HDFS says that it's for large streaming
> jobs, but I couldn't find an explicit answer to this, so I'll try
> asking here. How is HDFS's random seek performance within an
> FSDataInputStream? I use lucene with a lot of indices (potentially
> thousands), so I was thinking of putting them into HDFS and
> reimplementing my search as a Hadoop map-reduce. I've noticed that
> lucene tends to do a bit of random seeking when searching though; I
> don't believe that it guarantees that all seeks be to increasing file
> positions either.
> > Would HDFS be a bad fit for an access pattern that involves seeks to
> random positions within a stream?
> > Also, is getFileStatus the typical way of getting the length of a file
> in HDFS, or is there some method on FSDataInputStream that I'm not
> seeing?
> > Please cc: me on any reply; I'm not on the hadoop list. Thanks!

Re: Scaling out/up or a mix

jason hadoop — Sat, 27 Jun 2009 09:10:25 -0700

How about multi-threaded mappers?
Multi-Threaded mappers are ideal for map tasks that are non locally io bound
with many distinct endpoints.
You can also control the thread count on a per job basis. On Sat, Jun 27, 2009 at 8:26 AM, Marcus Herou <marcus.herou@tailsweep.com>wrote: > The argument currently against increasing num-mappers is that the machines
> will get into oom and since a lot of the jobs are crawlers I need more
> ip-numbers so I don't get banned :)
>
> Thing is that we currently have solr on the very same machines and
> data-nodes as well so I can only give the MR nodes about 1G memory since I
> need SOLR to have 4G...
>
> Now I see that I should get some obvious and juste critique about the
> layout
> of this arch but I'm a little limited in budget and so is then the arch :)
>
> However is it wise to have the MR tasks on the same nodes as the data-nodes
> or should I split the arch ? I mean the data-nodes perhaps need more
> disk-IO
> and the MR more memory and CPU ?
>
> Trying to find a sweetspot hardware spec of those two roles.
>
> //Marcus
>
>
>
> On Sat, Jun 27, 2009 at 4:24 AM, Brian Bockelman <bbockelm@cse.unl.edu
> >wrote:
>
> > Hey Marcus,
> >
> > Are you recording the data rates coming out of HDFS? Since you have such
> a
> > low CPU utilizations, I'd look at boxes utterly packed with big hard
> drives
> > (also, why are you using RAID1 for Hadoop??).
> >
> > You can get 1U boxes with 4 drive bays or 2U boxes with 12 drive bays.
> > Based on the data rates you see, make the call.
> >
> > On the other hand, what's the argument against running 3x more mappers
> per
> > box? It seems that your boxes still have more overhead to use -- there's
> no
> > I/O wait.
> >
> > Brian
> >
> >
> > On Jun 26, 2009, at 4:43 PM, Marcus Herou wrote:
> >
> > Hi.
> >>
> >> We have a deployment of 10 hadoop servers and I now need more mapping
> >> capability (no not just add more mappers per instance) since I have so
> >> many
> >> jobs running. Now I am wondering what I should aim on...
> >> Memory, cpu or disk... How long is a rope perhaps you would say ?
> >>
> >> A typical server is currently using about 15-20% cpu today on a
> quad-core
> >> 2.4Ghz 8GB RAM machine with 2 RAID1 SATA 500GB disks.
> >>
> >> Some specs below.
> >>
> >>> mpstat 2 5
> >>>
> >> Linux 2.6.24-19-server (mapreduce2) 06/26/2009
> >>
> >> 11:36:13 PM CPU %user %nice %sys %iowait %irq %soft %steal
> >> %idle intr/s
> >> 11:36:15 PM all 22.82 0.00 3.24 1.37 0.62 2.49 0.00
> >> 69.45 8572.50
> >> 11:36:17 PM all 13.56 0.00 1.74 1.99 0.62 2.61 0.00
> >> 79.48 8075.50
> >> 11:36:19 PM all 14.32 0.00 2.24 1.12 1.12 2.24 0.00
> >> 78.95 9219.00
> >> 11:36:21 PM all 14.71 0.00 0.87 1.62 0.25 1.75 0.00
> >> 80.80 8489.50
> >> 11:36:23 PM all 12.69 0.00 0.87 1.24 0.50 0.75 0.00
> >> 83.96 5495.00
> >> Average: all 15.62 0.00 1.79 1.47 0.62 1.97 0.00
> >> 78.53 7970.30
> >>
> >> What I am thinking is... Is it wiser to go for many of these cheap boxes
> >> with 8GB of RAM or should I for instance focus on machines which can
> give
> >> more I|O throughput ?
> >>
> >> I know that these things are hard but perhaps someone have draw some
> >> conclusions before the pragmatic way.
> >>
> >> Kindly
> >>
> >> //Marcus
> >>
> >>
> >> --
> >> Marcus Herou CTO and co-founder Tailsweep AB
> >> +46702561312
> >> marcus.herou@tailsweep.com
> >> http://www.tailsweep.com/
> >>
> >
> >
>
>
> --
> Marcus Herou CTO and co-founder Tailsweep AB
> +46702561312
> marcus.herou@tailsweep.com
> http://www.tailsweep.com/
> -- Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals

Re: Confused about partitioning and reducers

Stuart White — Sat, 27 Jun 2009 08:30:17 -0700

Please disregard this question. I think I'm mistaken. On Sat, Jun 27, 2009 at 10:25 AM, Stuart White <stuart.white1@gmail.com>wrote: > If I call HashPartitioner.getPartition(), passing a key of 4 and a
> numPartitions of 5, it returns a partition of 4. (Which is what I would
> expect.)
>
> However, if I have a mapred job, and in my mapper I emit a record with key
> 4, I'm configured to use the HashPartitioner, I have 5 Reducers configured,
> and I'm using the IdentityReducer, the record with key 4 gets handled by
> Reducer #0 (because it gets written out to part-00000).
>
> I would have expected a record with key 4 to be handled by reducer #4 (and
> therefore written to part-00004) because the HashPartitioner returns 4 for a
> key of 4 and a numPartitions of 5.
>
> Obviously I'm missing something here. What is the logic for deciding which
> partition of records is handled by which reducer instance?
>
> It can't be random, otherwise mapside join wouldn't work.
>
> Thanks.
>

Re: Scaling out/up or a mix

Marcus Herou — Sat, 27 Jun 2009 08:26:31 -0700

The argument currently against increasing num-mappers is that the machines
will get into oom and since a lot of the jobs are crawlers I need more
ip-numbers so I don't get banned :) Thing is that we currently have solr on the very same machines and
data-nodes as well so I can only give the MR nodes about 1G memory since I
need SOLR to have 4G... Now I see that I should get some obvious and juste critique about the layout
of this arch but I'm a little limited in budget and so is then the arch :) However is it wise to have the MR tasks on the same nodes as the data-nodes
or should I split the arch ? I mean the data-nodes perhaps need more disk-IO
and the MR more memory and CPU ? Trying to find a sweetspot hardware spec of those two roles. //Marcus On Sat, Jun 27, 2009 at 4:24 AM, Brian Bockelman <bbockelm@cse.unl.edu>wrote: > Hey Marcus,
>
> Are you recording the data rates coming out of HDFS? Since you have such a
> low CPU utilizations, I'd look at boxes utterly packed with big hard drives
> (also, why are you using RAID1 for Hadoop??).
>
> You can get 1U boxes with 4 drive bays or 2U boxes with 12 drive bays.
> Based on the data rates you see, make the call.
>
> On the other hand, what's the argument against running 3x more mappers per
> box? It seems that your boxes still have more overhead to use -- there's no
> I/O wait.
>
> Brian
>
>
> On Jun 26, 2009, at 4:43 PM, Marcus Herou wrote:
>
> Hi.
>>
>> We have a deployment of 10 hadoop servers and I now need more mapping
>> capability (no not just add more mappers per instance) since I have so
>> many
>> jobs running. Now I am wondering what I should aim on...
>> Memory, cpu or disk... How long is a rope perhaps you would say ?
>>
>> A typical server is currently using about 15-20% cpu today on a quad-core
>> 2.4Ghz 8GB RAM machine with 2 RAID1 SATA 500GB disks.
>>
>> Some specs below.
>>
>>> mpstat 2 5
>>>
>> Linux 2.6.24-19-server (mapreduce2) 06/26/2009
>>
>> 11:36:13 PM CPU %user %nice %sys %iowait %irq %soft %steal
>> %idle intr/s
>> 11:36:15 PM all 22.82 0.00 3.24 1.37 0.62 2.49 0.00
>> 69.45 8572.50
>> 11:36:17 PM all 13.56 0.00 1.74 1.99 0.62 2.61 0.00
>> 79.48 8075.50
>> 11:36:19 PM all 14.32 0.00 2.24 1.12 1.12 2.24 0.00
>> 78.95 9219.00
>> 11:36:21 PM all 14.71 0.00 0.87 1.62 0.25 1.75 0.00
>> 80.80 8489.50
>> 11:36:23 PM all 12.69 0.00 0.87 1.24 0.50 0.75 0.00
>> 83.96 5495.00
>> Average: all 15.62 0.00 1.79 1.47 0.62 1.97 0.00
>> 78.53 7970.30
>>
>> What I am thinking is... Is it wiser to go for many of these cheap boxes
>> with 8GB of RAM or should I for instance focus on machines which can give
>> more I|O throughput ?
>>
>> I know that these things are hard but perhaps someone have draw some
>> conclusions before the pragmatic way.
>>
>> Kindly
>>
>> //Marcus
>>
>>
>> --
>> Marcus Herou CTO and co-founder Tailsweep AB
>> +46702561312
>> marcus.herou@tailsweep.com
>> http://www.tailsweep.com/
>>
>
> -- Marcus Herou CTO and co-founder Tailsweep AB
+46702561312
marcus.herou@tailsweep.com
http://www.tailsweep.com/

Confused about partitioning and reducers

Stuart White — Sat, 27 Jun 2009 08:25:15 -0700

If I call HashPartitioner.getPartition(), passing a key of 4 and a
numPartitions of 5, it returns a partition of 4. (Which is what I would
expect.) However, if I have a mapred job, and in my mapper I emit a record with key
4, I'm configured to use the HashPartitioner, I have 5 Reducers configured,
and I'm using the IdentityReducer, the record with key 4 gets handled by
Reducer #0 (because it gets written out to part-00000). I would have expected a record with key 4 to be handled by reducer #4 (and
therefore written to part-00004) because the HashPartitioner returns 4 for a
key of 4 and a numPartitions of 5. Obviously I'm missing something here. What is the logic for deciding which
partition of records is handled by which reducer instance? It can't be random, otherwise mapside join wouldn't work. Thanks.

Archives not getting unarchived at tasktrackers

akhil1988 — Sat, 27 Jun 2009 06:36:40 -0700

 Hi All, I am using DistributedCache.addCacheArchives() to distribute a tar file to
the tasktrackers using the following statement. DistributedCache.addCacheArchives(new URI("/home/akhil1988/sample.tar"),
conf); According to the documentation it should get unarchived at the tasktrackers.
But the statement: DistributedCache.getLocalCacheArchives(conf); returns the following Path /hadoop/tmp/hadoop/mapred/local/taskTracker/archive/cn1.cloud.cs.illinois.edu/home/akhil1988/sample.tar That means sample.tar did not get unarchived.
Nor I am able to access file sample.txt in the above folder. Can anyone tell where I am going wrong? I tarred the file sample.txt using the following command: tar -cvf
sample.tar sample.txt Thanks,
Akhil
-- View this message in context: http://www.nabble.com/Archives-not-getting-unarchived-at-tasktrackers-tp24233281p24233281.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Re: Using the Stanford NLP with hadoop

hari939 — Sat, 27 Jun 2009 01:35:09 -0700

 by 'ClassName', which class are you actually refering to?
the class in which the LexicalParser is invoked? in my code, the class that implements the parser is named 'parse'
and this is the code that i used. lp = new LexicalizedParser(new ObjectInputStream(new
GZIPInputStream(parse.class.getResourceAsStream("/englishPCFG.ser.gz")))); the program runs to completion and map-reduce process is declared as
successfully completed everytime even if the code is changed to lp = new LexicalizedParser(new ObjectInputStream(new
GZIPInputStream(parse.class.getResourceAsStream("/englishPCF_G.ser.gz")))); this indicates that the getResourceAsStream does throw an exception even if
the file is not present, i guess. any ideas? :confused: Kevin Peterson-3 wrote:
> > On Sat, Apr 18, 2009 at 5:18 AM, hari939 wrote:
> >>
>> My project of parsing through material for a semantic search engine
>> requires
>> me to use the http://nlp.stanford.edu/software/lex-parser.shtml Stanford
>> NLP parser on hadoop cluster.
>>
>> To use the Stanford NLP parser, one must create a lexical parser object
>> using a englishPCFG.ser.gz file as a constructor's parameter.
>> i have tried loading the file onto the Hadoop dfs in the /user/root/
>> folder
>> and have also tried packing the file along with the jar of the java
>> program.
> > > Use getResourceAsStream to read it from the jar.
> > Use the ObjectInputStream constructor.
> > That is, new LexicalizedParser(new ObjectInputStream(new
> GzipInputStream(ClassName.class.getResourceAsStream("/englishPCFG.ser.gz")))
> > I'm interested to know if you have found any other open source parsers in
> Java or at least have java bindings.
> > -- View this message in context: http://www.nabble.com/Using-the-Stanford-NLP-with-hadoop-tp23112316p24231349.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Re: Scaling out/up or a mix

Brian Bockelman — Fri, 26 Jun 2009 19:24:31 -0700

Hey Marcus, Are you recording the data rates coming out of HDFS? Since you have such a low CPU utilizations, I'd look at boxes utterly packed with big hard drives (also, why are you using RAID1 for Hadoop??). You can get 1U boxes with 4 drive bays or 2U boxes with 12 drive bays. Based on the data rates you see, make the call. On the other hand, what's the argument against running 3x more mappers per box? It seems that your boxes still have more overhead to use -- there's no I/O wait. Brian On Jun 26, 2009, at 4:43 PM, Marcus Herou wrote: > Hi.
>
> We have a deployment of 10 hadoop servers and I now need more mapping
> capability (no not just add more mappers per instance) since I have > so many
> jobs running. Now I am wondering what I should aim on...
> Memory, cpu or disk... How long is a rope perhaps you would say ?
>
> A typical server is currently using about 15-20% cpu today on a quad- > core
> 2.4Ghz 8GB RAM machine with 2 RAID1 SATA 500GB disks.
>
> Some specs below.
>> mpstat 2 5
> Linux 2.6.24-19-server (mapreduce2) 06/26/2009
>
> 11:36:13 PM CPU %user %nice %sys %iowait %irq %soft > %steal
> %idle intr/s
> 11:36:15 PM all 22.82 0.00 3.24 1.37 0.62 2.49 > 0.00
> 69.45 8572.50
> 11:36:17 PM all 13.56 0.00 1.74 1.99 0.62 2.61 > 0.00
> 79.48 8075.50
> 11:36:19 PM all 14.32 0.00 2.24 1.12 1.12 2.24 > 0.00
> 78.95 9219.00
> 11:36:21 PM all 14.71 0.00 0.87 1.62 0.25 1.75 > 0.00
> 80.80 8489.50
> 11:36:23 PM all 12.69 0.00 0.87 1.24 0.50 0.75 > 0.00
> 83.96 5495.00
> Average: all 15.62 0.00 1.79 1.47 0.62 1.97 > 0.00
> 78.53 7970.30
>
> What I am thinking is... Is it wiser to go for many of these cheap > boxes
> with 8GB of RAM or should I for instance focus on machines which can > give
> more I|O throughput ?
>
> I know that these things are hard but perhaps someone have draw some
> conclusions before the pragmatic way.
>
> Kindly
>
> //Marcus
>
>
> -- > Marcus Herou CTO and co-founder Tailsweep AB
> +46702561312
> marcus.herou@tailsweep.com
> http://www.tailsweep.com/

Map/Reduce Errors

"Usman Waheed" — Fri, 26 Jun 2009 19:23:34 -0700

Hi All, I had posted a question earlier regarding some not so intuitive error messages that I was getting on one of the clusters when trying to map/reduce. After many hours of googling :) i found a post that solved my problem. http://www.mail-archive.com/core-user@hadoop.apache.org/msg07202.html. One of our engineers ran way too many jobs that created enormous subdirs in $HADOOP_HOME/logs/userlogs. Deleting these subdirs under $HADOOP_HOME/logs/userlogs/ on the datanodes solved the problem. You can also set the cleanup in the hadoop-default.xml file by setting the cleanup time to x hours instead of 24. The specific param is userlogs.retain. Just wanted to share this with you all. Thanks,
Usman -- Using Opera's revolutionary e-mail client: http://www.opera.com/mail/

Re: Hadoop0.20 - Class Not Found exception

imcaptor — Fri, 26 Jun 2009 18:58:25 -0700

I meet the question also, resolved by transfer a class name to JobConf
constructor. If you new the *JobConf,you must transfer a class name to it.* 2009/6/27 Amandeep Khurana <amansk@gmail.com> > I'm getting the following error while starting a MR job:
>
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
> oracle.jdbc.driver.OracleDriver
> at
>
> org.apache.hadoop.mapred.lib.db.DBInputFormat.configure(DBInputFormat.java:297)
> ... 21 more
> Caused by: java.lang.ClassNotFoundException:
> oracle.jdbc.driver.OracleDriver
> at java.net.URLClassLoader$1.run(Unknown Source)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(Unknown Source)
> at java.lang.ClassLoader.loadClass(Unknown Source)
> at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
> at java.lang.ClassLoader.loadClass(Unknown Source)
> at java.lang.ClassLoader.loadClassInternal(Unknown Source)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Unknown Source)
> at
>
> org.apache.hadoop.mapred.lib.db.DBConfiguration.getConnection(DBConfiguration.java:123)
> at
>
> org.apache.hadoop.mapred.lib.db.DBInputFormat.configure(DBInputFormat.java:292)
> ... 21 more
>
> Interestingly, the relevant jar is bundled into the MR job jar and its also
> there in the $HADOOP_HOME/lib directory.
>
> Exactly same thing worked with 0.19.. Not sure what could have changed or I
> broke to cause this error...
>
> Amandeep
>
>
> Amandeep Khurana
> Computer Science Graduate Student
> University of California, Santa Cruz
>

Re: Can I post pig questions on this forum?

Alan Gates — Fri, 26 Jun 2009 17:57:14 -0700

pig-user@hadoop.apache.org is the right place for pig questions. Alan. On Jun 26, 2009, at 4:34 PM, pmg wrote: >
>
> -- > View this message in context: http://www.nabble.com/Can-I-post-pig-questions-on-this-forum--tp24228728p24228728.html
> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
>

Re: Using addCacheArchive

akhil1988 — Fri, 26 Jun 2009 17:55:39 -0700

 Thanks Chris for your reply! Well, I could not understand much of what has been discussed on that forum.
I am unaware of Cascading. My problem is simple - I want a directory to present in the local working
directory of tasks so that I can access it from my map task in the following
manner : FileInputStream fin = new FileInputStream("Config/file1.config"); where,
Config is a directory which contains many files/directories, one of which is
file1.config It would be helpful to me if you can tell me what statements to use to
distribute a directory to the tasktrackers.
The API doc http://hadoop.apache.org/core/docs/r0.20.0/api/index.html says
that archives are unzipped on the tasktrackers but I want an example of how
to use this in case of a dreictory. Thanks,
Akhil Chris Curtin-2 wrote:
> > Hi,
> > I've found it much easier to write the file to HDFS use the API, then pass
> the 'path' to the file in HDFS as a property. You'll need to remember to
> clean up the file after you're done with it.
> > Example details are in this thread:
> http://groups.google.com/group/cascading-user/browse_thread/thread/d5c619349562a8d6#
> > Hope this helps,
> > Chris
> > On Thu, Jun 25, 2009 at 4:50 PM, akhil1988 <akhilanger@gmail.com> wrote:
> >>
>> Please ask any questions if I am not clear above about the problem I am
>> facing.
>>
>> Thanks,
>> Akhil
>>
>> akhil1988 wrote:
>> >
>> > Hi All!
>> >
>> > I want a directory to be present in the local working directory of the
>> > task for which I am using the following statements:
>> >
>> > DistributedCache.addCacheArchive(new URI("/home/akhil1988/Config.zip"),
>> > conf);
>> > DistributedCache.createSymlink(conf);
>> >
>> >>> Here Config is a directory which I have zipped and put at the given
>> >>> location in HDFS
>> >
>> > I have zipped the directory because the API doc of DistributedCache
>> > (http://hadoop.apache.org/core/docs/r0.20.0/api/index.html) says that
>> the
>> > archive files are unzipped in the local cache directory :
>> >
>> > DistributedCache can be used to distribute simple, read-only data/text
>> > files and/or more complex types such as archives, jars etc. Archives
>> (zip,
>> > tar and tgz/tar.gz files) are un-archived at the slave nodes.
>> >
>> > So, from my understanding of the API docs I expect that the Config.zip
>> > file will be unzipped to Config directory and since I have SymLinked
>> them
>> > I can access the directory in the following manner from my map
>> function:
>> >
>> > FileInputStream fin = new FileInputStream("Config/file1.config");
>> >
>> > But I get the FileNotFoundException on the execution of this statement.
>> > Please let me know where I am going wrong.
>> >
>> > Thanks,
>> > Akhil
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Using-addCacheArchive-tp24207739p24210836.html
>> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>>
>>
> > -- View this message in context: http://www.nabble.com/Using-addCacheArchive-tp24207739p24229338.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Re: FileStatus.getLen(): bug in documentation or bug in implememtation?

Konstantin Shvachko — Fri, 26 Jun 2009 16:51:11 -0700

Documentation is wrong. Implementation wins.
Could you please file a bug. Thanks,
--Konstantin Dima Rzhevskiy wrote:
> Hi all
> I try get length of file hadoop(RawFilesysten or hdfs) .
> In javadoc method org.apache.hadoop.fs.FileStatus.getLen() writtend that
> this method "return the length of this file, in blocks"
> But method return size in bytes.
> > Is this bug in documentation or implememtation?
> I use hadoop-0.18.3.
> > > Dmitry Rzhevskiy.
>

Hadoop0.20 - Class Not Found exception

Amandeep Khurana — Fri, 26 Jun 2009 16:43:04 -0700

I'm getting the following error while starting a MR job: Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
oracle.jdbc.driver.OracleDriver at
org.apache.hadoop.mapred.lib.db.DBInputFormat.configure(DBInputFormat.java:297) ... 21 more
Caused by: java.lang.ClassNotFoundException: oracle.jdbc.driver.OracleDriver at java.net.URLClassLoader$1.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClass(Unknown Source) at java.lang.ClassLoader.loadClassInternal(Unknown Source) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Unknown Source) at
org.apache.hadoop.mapred.lib.db.DBConfiguration.getConnection(DBConfiguration.java:123) at
org.apache.hadoop.mapred.lib.db.DBInputFormat.configure(DBInputFormat.java:292) ... 21 more Interestingly, the relevant jar is bundled into the MR job jar and its also
there in the $HADOOP_HOME/lib directory. Exactly same thing worked with 0.19.. Not sure what could have changed or I
broke to cause this error... Amandeep Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz

Re: Can I post pig questions on this forum?

Christophe Bisciglia — Fri, 26 Jun 2009 16:36:34 -0700

pig-user@hadoop.apache.org On Fri, Jun 26, 2009 at 4:34 PM, pmg<parmod.mehta@gmail.com> wrote:
>
>
> --
> View this message in context: http://www.nabble.com/Can-I-post-pig-questions-on-this-forum--tp24228728p24228728.html
> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
>
> -- get hadoop: cloudera.com/hadoop
online training: cloudera.com/hadoop-training
blog: cloudera.com/blog
twitter: twitter.com/cloudera

Can I post pig questions on this forum?

pmg — Fri, 26 Jun 2009 16:34:05 -0700

 -- View this message in context: http://www.nabble.com/Can-I-post-pig-questions-on-this-forum--tp24228728p24228728.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.

FileStatus.getLen(): bug in documentation or bug in implememtation?

Dima Rzhevskiy — Fri, 26 Jun 2009 16:18:51 -0700

Hi all
I try get length of file hadoop(RawFilesysten or hdfs) .
In javadoc method org.apache.hadoop.fs.FileStatus.getLen() writtend that
this method "return the length of this file, in blocks"
But method return size in bytes. Is this bug in documentation or implememtation?
I use hadoop-0.18.3. Dmitry Rzhevskiy.

Re: Permissions needed to run RandomWriter ?

Alex Loddengaard — Fri, 26 Jun 2009 16:06:28 -0700

Have you tried to run the example job as the superuser? It seems like this
might be an issue where hadoop.tmp.dir doesn't have the correctly
permissions. hadoop.tmp.dir and dfs.data.dir should be owned by the unix
user running your Hadoop daemons and owner-writtable and readable. Can you confirm this is the case? Thanks, Alex On Fri, Jun 26, 2009 at 1:29 PM, Mulcahy, Stephen
<stephen.mulcahy@deri.org>wrote: > [Apologies for the top-post, sending this from a dodgy webmail client]
>
> Hi Alex,
>
> My hadoop-site.xml is as follows,
>
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> <!-- Put site-specific property overrides in this file. -->
>
> <configuration>
> <property>
> <name>mapred.job.tracker</name>
> <value>hadoop01:9001</value>
> </property>
>
> <property>
> <name>fs.default.name</name>
> <value>hdfs://hadoop01:9000</value>
> </property>
>
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/data1/hadoop-tmp/</value>
> </property>
>
> <property>
> <name>dfs.data.dir</name>
> <value>/data1/hdfs,/data2/hdfs</value>
> </property>
> </configuration>
>
> Any comments welcome,
>
> -stephen
>
>
>
> -----Original Message-----
> From: Alex Loddengaard [mailto:alex@cloudera.com]
> Sent: Fri 26/06/2009 18:32
> To: core-user@hadoop.apache.org
> Subject: Re: Permissions needed to run RandomWriter ?
>
> Hey Stephen,
>
> What does your hadoop-site.xml look like? The Exception is in
> java.io.UnixFileSystem, which makes me think that you're actually creating
> and modifying directories on your local file system instead of HDFS. Make
> sure "fs.default.name" looks like "hdfs://your-namenode.domain.com:PORT".
>
> Alex
>
> On Fri, Jun 26, 2009 at 4:40 AM, stephen mulcahy
> <stephen.mulcahy@deri.org>wrote:
>
> > Hi,
> >
> > I've just installed a new test cluster and I'm trying to give it a quick
> > smoke test with RandomWriter and Sort.
> >
> > I can run these fine with the superuser account. When I try to run them
> as
> > another user I run into problems even though I've created the output
> > directory and given permissions to the other user to write to this
> > directory. i.e.
> >
> > 1. smulcahy@hadoop01:~$ hadoop fs -mkdir /foo
> > mkdir: org.apache.hadoop.fs.permission.AccessControlException: Permission
> > denied: user=smulcahy, access=WRITE, inode="":hadoop:supergroup:rwxr-xr-x
> >
> > OK - we don't have permissions anyways
> >
> > 2. hadoop@hadoop01:/$ hadoop fs -mkdir /foo
> >
> > OK
> >
> > 3. hadoop fs -chown -R smulcahy /foo
> >
> > OK
> >
> > 4. smulcahy@hadoop01:~$ hadoop fs -mkdir /foo/test
> >
> > OK
> >
> > 5. smulcahy@hadoop01:~$ hadoop jar /usr/lib/hadoop/hadoop-*-examples.jar
> > randomwriter /foo
> > java.io.IOException: Permission denied
> > at java.io.UnixFileSystem.createFileExclusively(Native Method)
> > at java.io.File.checkAndCreate(File.java:1704)
> > at java.io.File.createTempFile(File.java:1793)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:115)
> > at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> > at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
> >
> > Any suggestions on why step 5. is failing even though I have write
> > permissions to /foo - do I need permissions on some other directory also
> or
> > ... ?
> >
> > Thanks,
> >
> > -stephen
> >
> > --
> > Stephen Mulcahy, DI2, Digital Enterprise Research Institute,
> > NUI Galway, IDA Business Park, Lower Dangan, Galway, Ireland
> > http://di2.deri.ie http://webstar.deri.ie http://sindice.com
> >
>
>

difference between 'hadoop.tmp.dir' & 'mapred.temp.dir'

umer arshad — Fri, 26 Jun 2009 15:03:44 -0700

 Hi, Can some body kindly explain the difference b/w 'hadoop.tmp.dir' & 'mapred.temp.dir'.
I am trying to figure out where does the intermediate temporary files are stored for a mapreduce
job. Thanks,
--umer _________________________________________________________________
Invite your mail contacts to join your friends list with Windows Live Spaces. It's easy!
http://spaces.live.com/spacesapi.aspx?wx_action=create&wx_url=/friends.aspx&mkt=en-us

Re: Error in Cluster Startup: NameNode is not formatted

Matt Massie — Fri, 26 Jun 2009 14:45:13 -0700

The property "dfs.name.dir" allows you to control where Hadoop writes NameNode metadata. You should have a property like <property>
<name>dfs.name.dir</name>
<value>/data/zhang/hadoop/name/data</value>
</property> to make sure the NameNode data isn't being deleted when you delete the files in /tmp. -Matt On Jun 26, 2009, at 2:33 PM, Boyu Zhang wrote: > Matt,
>
> Thanks a lot for your reply! I did formatted the namenode. But I got > the
> same error again. And actually I successfully run the example jar > file once,
> but after that one time, I couldn't get it run again. I clean the / > tmp dir
> every time before I format namenode again(I am just testing it, so I > don't
> worry about losing data:). Still, I got the same error when I > execute the
> bin/start-dfs.sh . I checked my conf, and I can't figure out why. > Here is my
> conf file:
>
> I really appreciate if you could take a look at it. Thanks a lot.
>
>
> <configuration>
>
> <property>
> <name>fs.default.name</name>
> <value>hdfs://hostname1:9000</value>
> </property>
>
>
> <property>
> <name>mapred.job.tracker</name>
> <value>hostname2:9001</value>
> </property>
>
>
>
> <property>
> <name>dfs.data.dir</name>
> <value>/data/zhang/hadoop/dfs/data</value>
> <description>Determines where on the local filesystem an DFS data > node
> should store its blocks. If this is a comma-delimited
> list of directories, then data will be stored in all named
> directories, typically on different devices.
> Directories that do not exist are ignored.
> </description>
> </property>
>
>
> <property>
> <name>mapred.local.dir</name>
> <value>/data/zhang/hadoop/mapred/local</value>
> <description>The local directory where MapReduce stores intermediate
> data files. May be a comma-separated list of
> directories on different devices in order to spread disk i/o.
> Directories that do not exist are ignored.
> </description>
> </property>
> </configuration>
>
>
> -----Original Message-----
> From: Matt Massie [mailto:matt@cloudera.com]
> Sent: Friday, June 26, 2009 4:31 PM
> To: core-user@hadoop.apache.org
> Subject: Re: Error in Cluster Startup: NameNode is not formatted
>
> Boyu-
>
> You didn't do anything stupid. I've forgotten to format a NameNode
> too myself.
>
> If you check the QuickStart guide at
> http://hadoop.apache.org/core/docs/current/quickstart.html
> you'll see that formatting the NameNode is the first of the
> Execution section (near the bottom of the page).
>
> The command to format the NameNode is:
>
> hadoop namenode -format
>
> A warning though, you should only format your NameNode once. Just
> like formatting any filesystem, you can loss data if you (re)format.
>
> Good luck.
>
> -Matt
>
> On Jun 26, 2009, at 1:25 PM, Boyu Zhang wrote:
>
>> Hi all,
>>
>> I am a student and I am trying to install the Hadoop on a cluster, I
>> have
>> one machine running namenode, one running jobtracker, two slaves.
>>
>> When I run the /bin/start-dfs.sh , there is something wrong with my
>> namenode, it won't start. Here is the error message in the log file:
>>
>> ERROR org.apache.hadoop.fs.FSNamesystem: FSNamesystem initialization
>> failed.
>> java.io.IOException: NameNode is not formatted.
>> at
>> org.apache.hadoop.dfs.FSImage.recoverTransitionRead(FSImage.java:243)
>> at
>> org.apache.hadoop.dfs.FSDirectory.loadFSImage(FSDirectory.java:80)
>> at
>> org.apache.hadoop.dfs.FSNamesystem.initialize(FSNamesystem.java:294)
>> at
>> org.apache.hadoop.dfs.FSNamesystem.<init>(FSNamesystem.java:273)
>> at org.apache.hadoop.dfs.NameNode.initialize(NameNode.java:148)
>> at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:193)
>> at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:179)
>> at
>> org.apache.hadoop.dfs.NameNode.createNameNode(NameNode.java:830)
>> at org.apache.hadoop.dfs.NameNode.main(NameNode.java:839)
>>
>>
>> I think it is something stupid i did, could somebody help me out?
>> Thanks a
>> lot!
>>
>>
>> Sincerely,
>>
>> Boyu Zhang
>
>

Scaling out/up or a mix

Marcus Herou — Fri, 26 Jun 2009 14:43:42 -0700

Hi. We have a deployment of 10 hadoop servers and I now need more mapping
capability (no not just add more mappers per instance) since I have so many
jobs running. Now I am wondering what I should aim on...
Memory, cpu or disk... How long is a rope perhaps you would say ? A typical server is currently using about 15-20% cpu today on a quad-core
2.4Ghz 8GB RAM machine with 2 RAID1 SATA 500GB disks. Some specs below.
> mpstat 2 5
Linux 2.6.24-19-server (mapreduce2) 06/26/2009 11:36:13 PM CPU %user %nice %sys %iowait %irq %soft %steal
%idle intr/s
11:36:15 PM all 22.82 0.00 3.24 1.37 0.62 2.49 0.00
69.45 8572.50
11:36:17 PM all 13.56 0.00 1.74 1.99 0.62 2.61 0.00
79.48 8075.50
11:36:19 PM all 14.32 0.00 2.24 1.12 1.12 2.24 0.00
78.95 9219.00
11:36:21 PM all 14.71 0.00 0.87 1.62 0.25 1.75 0.00
80.80 8489.50
11:36:23 PM all 12.69 0.00 0.87 1.24 0.50 0.75 0.00
83.96 5495.00
Average: all 15.62 0.00 1.79 1.47 0.62 1.97 0.00
78.53 7970.30 What I am thinking is... Is it wiser to go for many of these cheap boxes
with 8GB of RAM or should I for instance focus on machines which can give
more I|O throughput ? I know that these things are hard but perhaps someone have draw some
conclusions before the pragmatic way. Kindly //Marcus -- Marcus Herou CTO and co-founder Tailsweep AB
+46702561312
marcus.herou@tailsweep.com
http://www.tailsweep.com/