Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-03-14
12:36:05 awalende Insufficient compute resources: Requested instance NUMA topology cannot fit the given host NUMA topology.
12:36:50 awalende Which is weird, I have setup hw:numa_nodes=4 in the flavor, since my Intel Xeon has 4 Nodes.
12:37:32 awalende agesTopology,NUMAPagesTopology,NUMAPagesTopology],pinned_cpus=set([]),siblings=[])
12:37:32 awalende beforeheand there is a bunch of repeating logs like: Attempting to fit instance cell InstanceNUMACell(cpu_pinning_raw=None,cpu_policy=None,cpu_thread_policy=None,cpu_topology=<?>,cpuset=set([0,1,2,3,4,5,6,7,8,9,10,11,12,13]),cpuset_reserved=None,id=0,memory=512,pagesize=256) on host_cell NUMACell(cpu_usage=0,cpuset=set([3,51,35,7,15,11,47,43,19,23,39,27,55,31]),id=3,memory=193532,memory_usage=0,mempages=[NUMAP
12:43:38 efried mdbooth: are you a NUMA guy?
12:43:51 efried awalende: (looking for someone who can spell NUMA)
12:43:58 awalende :D
12:46:50 openstackgerrit Jens Harbott (frickler) proposed openstack/nova master: Re-enable Ceph in live migration testing https://review.openstack.org/643150
12:55:49 openstackgerrit Elod Illes proposed openstack/nova stable/ocata: WIP: Use ubuntu-xenial for sphinx jobs https://review.openstack.org/643320
12:57:33 mdbooth efried: I am not, unfortunately.
12:58:01 efried kashyap: what about you? :)
12:58:33 mdbooth efried: I can spell it, though. Our 2 best numa guys are travelling today.
12:58:46 efried okay.
12:58:58 efried eventually here I'm gonna need to learn something about it...
12:59:10 kashyap efried: Hi, I have a NUMA-capable machine, and occasionally do testing of NUMA+SR-IOV stuff; besides that I am not an expert
12:59:39 kashyap efried: What's the question, it might make curious enough to learn the answer :-)
13:00:33 efried kashyap: awalende is having trouble scheduling an instance requesting 4 numa nodes and hugepages, see logs starting around :10
13:00:57 efried I'm guessing it's a simple thing - for somebody with a clue how NUMA-related scheduling works
13:01:19 efried http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-03-14.log.html#t2019-03-14T12:10:17
13:04:51 kashyap Hmm; the error "Requested instance NUMA topology cannot fit the given host NUMA topology" is indeed weird *if* you have told the scheduler to use more than one NUMA node
13:04:54 kashyap ... which you did.
13:06:59 mdbooth kashyap: Did you rebase that writeback patch for some reason?
13:08:18 mdbooth Incidentally, ^^^ is a super-obvious change in light of the informative comment kashyap has added to disk_cachemode()
13:08:37 mdbooth Erm... https://review.openstack.org/#/c/641981/
13:09:07 kashyap mdbooth: Had to rebase because "add the compress flag" thingie to 'qemu-img' got merged (you'll recall it, as you reviewed it)
13:09:17 mdbooth kashyap: Ah, ok.
13:12:02 kashyap (It caused a conflict, so I was forced to fix it and rebase :-))
13:15:42 kashyap mdbooth: What about also doing: s/writethrough/writeback/ for SMBFS and virtuozzo?
13:15:50 kashyap See: nova/virt/libvirt/volume/smbfs.py and nova/virt/libvirt/volume/vzstorage.py.
13:16:04 mdbooth kashyap: We should do that.
13:16:14 mdbooth kashyap: Separate patches though, please.
13:16:18 kashyap Yes!
13:16:22 kashyap That's what I noted in the TODO
13:16:27 kashyap (In the commit)
13:16:27 mdbooth Also super-low priority, because likely nobody cares.
13:16:36 kashyap [quote]
13:16:37 kashyap TODO: Investigate if this is okay for Cinder volume drivers such as SMBFS and Virtuozzo. If so, then change them to use 'writeback' as well (probably in separate patches).
13:16:40 kashyap [/quote]
13:16:52 kashyap mdbooth: Yes, nobody cares, except when it comes to bite you.
13:17:04 kashyap Might as well do it when I remember it; or else it goes into /dev/null of my brain
13:17:06 mdbooth kashyap: It would only bite you if you used those things.
13:17:24 kashyap (Yep.)
13:28:17 openstackgerrit Boxiang Zhu proposed openstack/nova master: Raise error when force_raw_images and images_type are incompatible https://review.openstack.org/640271
13:34:45 openstackgerrit Boxiang Zhu proposed openstack/nova master: Remove unsafe convert of lvm imagebackend https://review.openstack.org/642667
13:37:13 zhubx007 mdbooth: looking forward to your review on two patches ^ thanks :)
13:45:03 mriedem dansmith: can i get your +2 back on https://review.openstack.org/#/c/643151/ - i had to fix a test otherwise it's the same
13:45:21 dansmith mriedem: well, the test failed in a weird way in functional,
13:45:30 dansmith so I was waiting to see how the recheck turned out
13:45:38 mriedem that's the infra changes
13:45:39 mriedem but ok
13:45:51 mriedem this https://review.openstack.org/#/c/643129/
13:45:52 dansmith I looked in the log and couldn't see anything actually broken
13:46:00 mriedem it was this https://review.openstack.org/#/c/643129/4/.zuul.yaml
13:46:04 mriedem broke everything over night
13:46:18 dansmith okay... and the result is it just doesn't even run?
13:46:36 mriedem the old func py35 job couldn't find the py35 interpreter
13:46:47 mriedem with the new job parent it should be setup for that
13:46:54 mriedem uses a xenial rather than bionic node
13:47:08 dansmith okay I didn't see anything about that in the log, but that's fine, just sayin' I was looking at it already this morning and that's why I didn't just slap back on there
13:47:13 dansmith but if it's known
13:47:14 dansmith ..
13:47:37 mriedem thanks
13:52:21 mriedem frickler: looks like you fixed that ceph thing https://review.openstack.org/#/c/643150/
13:53:14 openstackgerrit Eric Fried proposed openstack/nova master: WIP/PoC: Bypass ironicclient for node.get https://review.openstack.org/642899
14:10:56 openstackgerrit Matt Riedemann proposed openstack/nova master: Re-enable Ceph in live migration testing https://review.openstack.org/643150
14:10:57 openstackgerrit Matt Riedemann proposed openstack/nova master: Customize irrelevant-files for nova-live-migration job https://review.openstack.org/643346
14:18:51 openstackgerrit Elod Illes proposed openstack/nova stable/ocata: Use ubuntu-xenial for sphinx jobs https://review.openstack.org/643320
14:40:57 dansmith is jaypipes just never around anymore?
14:46:40 cdent dansmith: I _think_ he's travelling this week. Not sure if it was business or pleasure.
14:46:48 dansmith oh okay
14:47:09 cdent he was at least on a plane on monday
15:41:22 kashyap aspiers: When you're about, refer to comment#11 here: https://review.openstack.org/#/c/633855/11
15:47:42 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: qemu: Make disk image conversion dramatically faster https://review.openstack.org/640781
15:47:43 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: vzstorage: Use 'writeback' QEMU cache mode https://review.openstack.org/643376
15:47:43 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: Use 'writeback' QEMU cache mode for Nova instances https://review.openstack.org/641981
15:47:44 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: smbfs: Use 'writeback' QEMU cache mode https://review.openstack.org/643377
15:54:42 cdent I
15:57:57 edleafe cdent: Roman numerals, anyone?
16:00:05 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: vzstorage: Use 'writeback' QEMU cache mode https://review.openstack.org/643376
16:00:06 openstackgerrit Kashyap Chamarthy proposed openstack/nova master: libvirt: smbfs: Use 'writeback' QEMU cache mode https://review.openstack.org/643377
16:00:28 kashyap Some simple catch above, if anyone wants a change of scenery.
16:00:47 kashyap (The core change and documentation is in: libvirt: Use 'writeback' QEMU cache mode for Nova instances https://review.openstack.org/641981)
16:02:14 cdent edleafe: α ω
16:02:57 edleafe cdent: make up your mind: greek or roman??
16:04:48 cfriesen kashyap: quick question. did you try running a "sync" after the writeback image conversion test to ensure the data had actually hit the disk?
16:05:17 cfriesen or do we even care if it did or not?
16:05:49 kashyap cfriesen: No, I didn't do. But, 'sync' is not required at all in this case -- why?
16:06:59 cfriesen well, your numbers look a lot like it just wrote to the cache. If the host crashes at that point, it might think it has valid data on-disk but actually have a corrupt image.
16:09:35 cfriesen now this may not actually be a problem in practice, just thought I'd raise it.
16:11:10 kashyap cfriesen: I got dropped off my IRC proxy; what was the last message you got from me?
16:11:31 kashyap (And if you wrote anything else after your "quick question ...", please re-post)
16:12:35 cfriesen I just said... your numbers look a lot like it just wrote to the cache. If the host crashes at that point, it might think it has valid data on-disk but actually have a corrupt image. Now this might not be a problem in practice, but I thought I'd mention it.
16:14:08 kashyap cfriesen: Noted, but ... I spent careful 4 days talking to QEMU Block Layer maint folks.
16:14:18 kashyap 17:05 < kashyap> cfriesen: No, I didn't do. But, 'sync' is not required at all in this case -- why?
16:14:18 kashyap And not sure if my messages got through:
16:14:21 kashyap 17:06 < kashyap> cfriesen: After image conversion completes successfully, QEMU issues fsync(), thus flushing the content down to disk.
16:14:38 kashyap cfriesen: So, the numbers are correct, FWIW. The data _is_ on disk.
16:15:05 kashyap cfriesen: I even audited QEMU's code (with call trac) to track down 'qemu-img convert' ends up calling fsync().
16:15:35 kashyap cfriesen: There is the call trace (since I sometimes see you pop into #qemu on OFTC :-)) -- https://kashyapc.fedorapeople.org/qemu-img-call-trace-to-fsync.txt
16:16:13 kashyap dansmith: Hehe, saw your "church retreat" remark on the channel archives (my IRC proxy kicked me, thanks to the IDENTIFY)

Earlier   Later