Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-03-14
11:23:26 kashyap (You were already +1 on those before.)
11:24:08 mdbooth "But that's not correct, because of our misunderstanding of how cache modes work."
11:24:23 mdbooth I'll say again: we never understood how cache modes work, only what they're called ;)
11:24:34 mdbooth ...misunderstood...
11:24:47 mdbooth Don't change that, though.
11:25:07 kashyap Hehe, noted.
11:25:38 kashyap Now I think not in terms of just named cache modes, but the three booleans each one impacts
11:27:22 kashyap mdbooth: FWIW, the little blob of text written in driver.py -- I got it reviewed by Kevin (of QEMU) over a few emails. So it's not me just riffing on it :-)
11:37:16 mdbooth ack
11:52:11 openstackgerrit Fan Zhang proposed openstack/nova master: Retry after hitting libvirt error VIR_ERR_OPERATION_INVALID in live migration. https://review.openstack.org/612272
12:10:17 awalende I'am trying to use hugepages with nova and qemu/libvirt. Hugepages is enabled and I have reserved a bunch of hugepages in standard 2048kb size. I added flavor properties with "flavor set hugePagesTest --property hw:mem_page_size=2Mb". However starting instances fails with "Exceeded maximum number of retries". Nova-compute log does not show any exceptions. Any Idea how to debug this?
12:34:57 efried awalende: Have you looked at the scheduler log?
12:36:04 awalende Yes, I could narrow it down. It has something to do with NUMA-Topology:
12:36:05 awalende Insufficient compute resources: Requested instance NUMA topology cannot fit the given host NUMA topology.
12:36:50 awalende Which is weird, I have setup hw:numa_nodes=4 in the flavor, since my Intel Xeon has 4 Nodes.
12:37:32 awalende beforeheand there is a bunch of repeating logs like: Attempting to fit instance cell InstanceNUMACell(cpu_pinning_raw=None,cpu_policy=None,cpu_thread_policy=None,cpu_topology=<?>,cpuset=set([0,1,2,3,4,5,6,7,8,9,10,11,12,13]),cpuset_reserved=None,id=0,memory=512,pagesize=256) on host_cell NUMACell(cpu_usage=0,cpuset=set([3,51,35,7,15,11,47,43,19,23,39,27,55,31]),id=3,memory=193532,memory_usage=0,mempages=[NUMAP
12:37:32 awalende agesTopology,NUMAPagesTopology,NUMAPagesTopology],pinned_cpus=set([]),siblings=[])
12:43:38 efried mdbooth: are you a NUMA guy?
12:43:51 efried awalende: (looking for someone who can spell NUMA)
12:43:58 awalende :D
12:46:50 openstackgerrit Jens Harbott (frickler) proposed openstack/nova master: Re-enable Ceph in live migration testing https://review.openstack.org/643150
12:55:49 openstackgerrit Elod Illes proposed openstack/nova stable/ocata: WIP: Use ubuntu-xenial for sphinx jobs https://review.openstack.org/643320
12:57:33 mdbooth efried: I am not, unfortunately.
12:58:01 efried kashyap: what about you? :)
12:58:33 mdbooth efried: I can spell it, though. Our 2 best numa guys are travelling today.
12:58:46 efried okay.
12:58:58 efried eventually here I'm gonna need to learn something about it...
12:59:10 kashyap efried: Hi, I have a NUMA-capable machine, and occasionally do testing of NUMA+SR-IOV stuff; besides that I am not an expert
12:59:39 kashyap efried: What's the question, it might make curious enough to learn the answer :-)
13:00:33 efried kashyap: awalende is having trouble scheduling an instance requesting 4 numa nodes and hugepages, see logs starting around :10
13:00:57 efried I'm guessing it's a simple thing - for somebody with a clue how NUMA-related scheduling works
13:01:19 efried http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-03-14.log.html#t2019-03-14T12:10:17
13:04:51 kashyap Hmm; the error "Requested instance NUMA topology cannot fit the given host NUMA topology" is indeed weird *if* you have told the scheduler to use more than one NUMA node
13:04:54 kashyap ... which you did.
13:06:59 mdbooth kashyap: Did you rebase that writeback patch for some reason?
13:08:18 mdbooth Incidentally, ^^^ is a super-obvious change in light of the informative comment kashyap has added to disk_cachemode()
13:08:37 mdbooth Erm... https://review.openstack.org/#/c/641981/
13:09:07 kashyap mdbooth: Had to rebase because "add the compress flag" thingie to 'qemu-img' got merged (you'll recall it, as you reviewed it)
13:09:17 mdbooth kashyap: Ah, ok.
13:12:02 kashyap (It caused a conflict, so I was forced to fix it and rebase :-))
13:15:42 kashyap mdbooth: What about also doing: s/writethrough/writeback/ for SMBFS and virtuozzo?
13:15:50 kashyap See: nova/virt/libvirt/volume/smbfs.py and nova/virt/libvirt/volume/vzstorage.py.
13:16:04 mdbooth kashyap: We should do that.
13:16:14 mdbooth kashyap: Separate patches though, please.
13:16:18 kashyap Yes!
13:16:22 kashyap That's what I noted in the TODO
13:16:27 mdbooth Also super-low priority, because likely nobody cares.
13:16:27 kashyap (In the commit)
13:16:36 kashyap [quote]
13:16:37 kashyap TODO: Investigate if this is okay for Cinder volume drivers such as SMBFS and Virtuozzo. If so, then change them to use 'writeback' as well (probably in separate patches).
13:16:40 kashyap [/quote]
13:16:52 kashyap mdbooth: Yes, nobody cares, except when it comes to bite you.
13:17:04 kashyap Might as well do it when I remember it; or else it goes into /dev/null of my brain
13:17:06 mdbooth kashyap: It would only bite you if you used those things.
13:17:24 kashyap (Yep.)
13:28:17 openstackgerrit Boxiang Zhu proposed openstack/nova master: Raise error when force_raw_images and images_type are incompatible https://review.openstack.org/640271
13:34:45 openstackgerrit Boxiang Zhu proposed openstack/nova master: Remove unsafe convert of lvm imagebackend https://review.openstack.org/642667
13:37:13 zhubx007 mdbooth: looking forward to your review on two patches ^ thanks :)
13:45:03 mriedem dansmith: can i get your +2 back on https://review.openstack.org/#/c/643151/ - i had to fix a test otherwise it's the same
13:45:21 dansmith mriedem: well, the test failed in a weird way in functional,
13:45:30 dansmith so I was waiting to see how the recheck turned out
13:45:38 mriedem that's the infra changes
13:45:39 mriedem but ok
13:45:51 mriedem this https://review.openstack.org/#/c/643129/
13:45:52 dansmith I looked in the log and couldn't see anything actually broken
13:46:00 mriedem it was this https://review.openstack.org/#/c/643129/4/.zuul.yaml
13:46:04 mriedem broke everything over night
13:46:18 dansmith okay... and the result is it just doesn't even run?
13:46:36 mriedem the old func py35 job couldn't find the py35 interpreter
13:46:47 mriedem with the new job parent it should be setup for that
13:46:54 mriedem uses a xenial rather than bionic node
13:47:08 dansmith okay I didn't see anything about that in the log, but that's fine, just sayin' I was looking at it already this morning and that's why I didn't just slap back on there
13:47:13 dansmith but if it's known
13:47:14 dansmith ..
13:47:37 mriedem thanks
13:52:21 mriedem frickler: looks like you fixed that ceph thing https://review.openstack.org/#/c/643150/
13:53:14 openstackgerrit Eric Fried proposed openstack/nova master: WIP/PoC: Bypass ironicclient for node.get https://review.openstack.org/642899
14:10:56 openstackgerrit Matt Riedemann proposed openstack/nova master: Re-enable Ceph in live migration testing https://review.openstack.org/643150
14:10:57 openstackgerrit Matt Riedemann proposed openstack/nova master: Customize irrelevant-files for nova-live-migration job https://review.openstack.org/643346
14:18:51 openstackgerrit Elod Illes proposed openstack/nova stable/ocata: Use ubuntu-xenial for sphinx jobs https://review.openstack.org/643320
14:40:57 dansmith is jaypipes just never around anymore?
14:46:40 cdent dansmith: I _think_ he's travelling this week. Not sure if it was business or pleasure.
14:46:48 dansmith oh okay
14:47:09 cdent he was at least on a plane on monday
15:41:22 kashyap aspiers: When you're about, refer to comment#11 here: https://review.openstack.org/#/c/633855/11
15:47:42 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: qemu: Make disk image conversion dramatically faster https://review.openstack.org/640781
15:47:43 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: Use 'writeback' QEMU cache mode for Nova instances https://review.openstack.org/641981
15:47:43 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: vzstorage: Use 'writeback' QEMU cache mode https://review.openstack.org/643376
15:47:44 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: smbfs: Use 'writeback' QEMU cache mode https://review.openstack.org/643377
15:54:42 cdent I
15:57:57 edleafe cdent: Roman numerals, anyone?
16:00:05 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: vzstorage: Use 'writeback' QEMU cache mode https://review.openstack.org/643376
16:00:06 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: smbfs: Use 'writeback' QEMU cache mode https://review.openstack.org/643377
16:00:28 kashyap Some simple catch above, if anyone wants a change of scenery.
16:00:47 kashyap (The core change and documentation is in: libvirt: Use 'writeback' QEMU cache mode for Nova instances https://review.openstack.org/641981)
16:02:14 cdent edleafe: α ω
16:02:57 edleafe cdent: make up your mind: greek or roman??
16:04:48 cfriesen kashyap: quick question. did you try running a "sync" after the writeback image conversion test to ensure the data had actually hit the disk?
16:05:17 cfriesen or do we even care if it did or not?
16:05:49 kashyap cfriesen: No, I didn't do. But, 'sync' is not required at all in this case -- why?

Earlier   Later