Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-12-03
20:32:48 dansmith some duplication gives us a bit of a backup in the case of cell downage as I said
20:33:00 artom Fair enough
20:33:39 artom OK, gives me stuff to think about
20:33:59 dansmith mriedem: doesn't pci_requests in the instance get fleshed out more than in the reqspec too?
20:34:59 artom I'll also need to think about backportability
20:35:13 mriedem maybe, don't know off the top of my head
20:35:17 artom *RH
20:35:17 openstack bugzilla.redhat.com bug 1715240 in openstack-nova "Resize ignores mem_page_size in new flavor" [High,On_dev] - Assigned to alifshit
20:35:17 artom Because while https://bugzilla.redhat.com/show_bug.cgi?id=1715240 is Newton is at last EOL for HR
20:35:36 mriedem i want to say that's Instance.pci_devices which is the allocated devices on the node
20:35:38 artom We'll probably want it in Queens as well
20:36:03 artom Need to bounce early to pick up 1/2 of my kids
20:36:06 artom Thank you gentlemen
20:36:12 dansmith mriedem: I thought we updated some physnet stuff in instance_pci_requests at least, but anyway, doesn't matter
20:36:21 mriedem what happened to the other half of the children?
20:36:36 artom mriedem, I have the other half of my marriage for that :D
20:37:05 dansmith I assume that when you get divorced and you have an even number of children, you just split those like the bank and other assets/liabilities right?
20:38:04 artom No, the woman takes everything
20:38:08 dansmith oh, okay
20:38:24 artom And you're just left with alcoholism and depresseion
20:38:42 artom (And those are not the names of the kids, btw)
20:38:45 dansmith haha
20:38:45 dansmith assuming those aren't the names of half your children.. gotcha
20:39:08 artom :D
21:01:03 openstackgerrit Matt Riedemann proposed openstack/nova master: ironic: report a custom trait for the node owner https://review.opendev.org/697184
21:16:14 eandersson capacity. ", "title": "Conflict"}]})
21:16:14 eandersson > Unable to submit allocation for instance x (409 {"errors": [{"status": 409, "request_id": "reqz", "code": "placement.undefined_code", "detail": "There was a conflict when trying to complete your request.\n\n Unable to allocate inventory: Unable to create allocation for 'VCPU' on resource provider y'. The requested amount would exceed the
21:16:23 eandersson This was the cause of the latest instance stuck in BUILD
21:17:51 mriedem those instances should all be buried in cell0 in conductor
21:17:56 mriedem scheduler should raise a NoValidHost
21:18:04 mnaser does setting `hw:mem_page_size` = `any` enable memory tracking on the host inside nova? reading the code leads to show that it does
21:18:28 mriedem eandersson: NoValidHost should be handled by conductor here https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1418
21:18:35 mnaser i'm running into an issue where doing numa node pinning without `hw:mem_page_size` yields in nova packing too many instances into numa node 0, which then results in `oom-killer`
21:19:06 eandersson I see one more log line on that req-id in the scheduler
21:19:07 eandersson > Computed NUMA topology CPU pinning: usable pCPUs: [[18, 38]], vCPUs mapping: [(0, 18), (1, 38)]
21:19:15 mriedem eandersson: and all of those build requests should result in creating instances in ERROR status in cell0 https://github.com/openstack/nova/blob/1c2b7d8f01814adfd6d28b97013a40cca51dfbdf/nova/conductor/manager.py#L1348
21:19:39 mriedem check your conductor logs
21:19:55 eandersson conductor logs are empty
21:19:59 sean-k-mooney mnaser: just got back. hw:mem_page_size=any was intened to allow the image to choose if hugepage or small pages should be used
21:20:18 sean-k-mooney mnaser: i belive it should enable the numa tracking but its the one policy that is least used
21:20:23 mnaser sean-k-mooney: but what if i dont want neither and i just want nova to be aware of memory per numa node?
21:20:39 mnaser cause that involves rebooting the machine and what not to enable those
21:20:51 eandersson and when I say empty, there are literately no log lines over the last 12 hours.
21:21:10 mnaser sean-k-mooney: according to virsh capabilities, the cell only has pages with size='4'
21:21:12 sean-k-mooney well you cant change an exitisng image to any with out a resize
21:21:24 mnaser right but for newly booted instances flavor change
21:21:48 mnaser update th flavor extra_specs and newly booted instances will do the right thing(tm)
21:21:50 sean-k-mooney right so hw:mem_page_size=small would use 4k pages
21:22:18 eandersson btw placement failed 6 times with the same erroor before it pretty much just stopped trying
21:22:19 sean-k-mooney any should also use 4k pages but would allow the image to request 2mb
21:22:42 mnaser sean-k-mooney: ok cool, i see, so 'any' should technically make things work and make nova start tracking memory
21:22:55 eandersson Its probably RabbitMQ issues again, but we tested all queues and binding and none are failing.
21:22:58 eandersson Like we have seen in the past.
21:23:15 sean-k-mooney i need to triple check it but if hw:mem_page_size is defiend it is ment to enable the numa aware tracking
21:23:27 sean-k-mooney no mater what value you set it to
21:23:31 mnaser sean-k-mooney: ok ill check and report back, but my notes seem to add up to yours :>
21:23:53 eandersson The most frustrating part is that there are no logs, and nothing in the database, so troubleshooting these usually requires looking into the database.
21:24:04 eandersson *nothing from the api
21:24:07 mnaser eandersson: did you have any rabbitmq outage/issue?
21:24:19 sean-k-mooney the reason im a little less confident with hw:mem_page_size=any is i know we dont have tempest testing for it
21:24:29 sean-k-mooney where as we did for large and small
21:24:47 eandersson mnaser in the past yea and we found bad bindings after that
21:24:48 mnaser in this case the only reason i care about mem page size is to get nova to track memory so maybe ill try small after all
21:24:59 eandersson we found a way to diagnose that
21:25:10 mnaser eandersson: in my experience anytime rabbitmq suffers .. anything .. you end up in that weird state and you have to restart the cloud
21:25:11 eandersson by pushing a fake message to the compute queues
21:25:15 mnaser ah gotcha
21:25:30 eandersson of course it might be something new with rmq
21:25:42 eandersson I know there is a patch to add mandatory to nova
21:25:52 eandersson the mandatory flag when publishing messages
21:26:23 eandersson Since if a message is lost you are stuck in buidling forever
21:26:29 eandersson with no error
21:27:01 mnaser my favorite :)
21:33:32 eandersson Gonna wipe the rmq db and see if it goes away
21:34:05 eandersson It's just odd that our synth tests are not hitting this
21:34:22 eandersson We create 300 VMs per day and none end up in this state
21:49:01 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Add node owner pre-filter https://review.opendev.org/697187
21:54:31 mriedem dansmith: this is probably a blast from the past for you https://review.opendev.org/#/c/697122/
21:54:33 mriedem mitaka era
22:09:58 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Implement cleanup_instance_network_on_host for neutron API https://review.opendev.org/697162
22:11:32 dansmith mriedem_away: I don't remember it, but I totes believe I did something awesome that someone else broke
23:00:51 sean-k-mooney eandersson: the patch for the mandatory flag i think is more or less stalled i have not seen anything happen on the fron in a while
23:01:37 eandersson Awh that is too bad
23:02:07 sean-k-mooney it looks like all the oslo.messaging fixes are done https://review.opendev.org/#/q/status:merged+project:openstack/oslo.messaging+branch:master+topic:bp/transport-options
23:02:25 sean-k-mooney but i dont think anyone has gotten around to using that in nova yet
23:02:43 eandersson Yea - and would probably not be easy to backport.
23:02:45 eandersson Or even possible.
23:03:33 sean-k-mooney well we need to do too things. frist start using it in master and second we need to figure out what the correct action is to take if we get an exception indicating it could not be delivered
23:03:44 eandersson I think I am going to add a log to nova just before it sends that notification to the computes.
23:05:01 sean-k-mooney for example if we could not deliver a message to a compute queue should we disabel that compute node so we dont try to schdule to it agian?
23:05:09 sean-k-mooney shoudl we jsut retry?
23:06:30 sean-k-mooney i actully have no idea what the best path forward would be in that case but we kind of need to figure that out before we can fix the issue
23:06:48 sean-k-mooney we now have the feature that allows use to adress it however
23:11:24 openstack Launchpad bug 1661510 in oslo.messaging "topic_send may loss messages if the queue not exists" [Medium,In progress] - Assigned to Gabriele Santomaggio (gsantomaggio)
23:11:24 sean-k-mooney melwitt: did you ever have time to talk to the oslo folks on how we could use this https://blueprints.launchpad.net/oslo.messaging/+spec/transport-options and specificly the mandaroty flag so we can adress the nova aspect of https://bugs.launchpad.net/oslo.messaging/+bug/1661510
23:12:05 sean-k-mooney i know i havent but it looks like all the patchs are in place on the oslo side
23:28:23 melwitt sean-k-mooney: I haven't talked to them bc like you said, their side is done. it's up to us (me) to figure out how to use it on the nova side and I haven't gotten a chance to dig in to it yet
23:28:54 melwitt I don't yet know where/how to pass it in the nova code
23:29:51 sean-k-mooney ya i just asked on the bug if they could provdie an example
23:30:51 sean-k-mooney i know its something we shoudl be setting when we do a topic send which is i guess a call to a compute node primarly

Earlier   Later