Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-12-03
21:20:51 eandersson and when I say empty, there are literately no log lines over the last 12 hours.
21:21:10 mnaser sean-k-mooney: according to virsh capabilities, the cell only has pages with size='4'
21:21:12 sean-k-mooney well you cant change an exitisng image to any with out a resize
21:21:24 mnaser right but for newly booted instances flavor change
21:21:48 mnaser update th flavor extra_specs and newly booted instances will do the right thing(tm)
21:21:50 sean-k-mooney right so hw:mem_page_size=small would use 4k pages
21:22:18 eandersson btw placement failed 6 times with the same erroor before it pretty much just stopped trying
21:22:19 sean-k-mooney any should also use 4k pages but would allow the image to request 2mb
21:22:42 mnaser sean-k-mooney: ok cool, i see, so 'any' should technically make things work and make nova start tracking memory
21:22:55 eandersson Its probably RabbitMQ issues again, but we tested all queues and binding and none are failing.
21:22:58 eandersson Like we have seen in the past.
21:23:15 sean-k-mooney i need to triple check it but if hw:mem_page_size is defiend it is ment to enable the numa aware tracking
21:23:27 sean-k-mooney no mater what value you set it to
21:23:31 mnaser sean-k-mooney: ok ill check and report back, but my notes seem to add up to yours :>
21:23:53 eandersson The most frustrating part is that there are no logs, and nothing in the database, so troubleshooting these usually requires looking into the database.
21:24:04 eandersson *nothing from the api
21:24:07 mnaser eandersson: did you have any rabbitmq outage/issue?
21:24:19 sean-k-mooney the reason im a little less confident with hw:mem_page_size=any is i know we dont have tempest testing for it
21:24:29 sean-k-mooney where as we did for large and small
21:24:47 eandersson mnaser in the past yea and we found bad bindings after that
21:24:48 mnaser in this case the only reason i care about mem page size is to get nova to track memory so maybe ill try small after all
21:24:59 eandersson we found a way to diagnose that
21:25:10 mnaser eandersson: in my experience anytime rabbitmq suffers .. anything .. you end up in that weird state and you have to restart the cloud
21:25:11 eandersson by pushing a fake message to the compute queues
21:25:15 mnaser ah gotcha
21:25:30 eandersson of course it might be something new with rmq
21:25:42 eandersson I know there is a patch to add mandatory to nova
21:25:52 eandersson the mandatory flag when publishing messages
21:26:23 eandersson Since if a message is lost you are stuck in buidling forever
21:26:29 eandersson with no error
21:27:01 mnaser my favorite :)
21:33:32 eandersson Gonna wipe the rmq db and see if it goes away
21:34:05 eandersson It's just odd that our synth tests are not hitting this
21:34:22 eandersson We create 300 VMs per day and none end up in this state
21:49:01 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Add node owner pre-filter https://review.opendev.org/697187
21:54:31 mriedem dansmith: this is probably a blast from the past for you https://review.opendev.org/#/c/697122/
21:54:33 mriedem mitaka era
22:09:58 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Implement cleanup_instance_network_on_host for neutron API https://review.opendev.org/697162
22:11:32 dansmith mriedem_away: I don't remember it, but I totes believe I did something awesome that someone else broke
23:00:51 sean-k-mooney eandersson: the patch for the mandatory flag i think is more or less stalled i have not seen anything happen on the fron in a while
23:01:37 eandersson Awh that is too bad
23:02:07 sean-k-mooney it looks like all the oslo.messaging fixes are done https://review.opendev.org/#/q/status:merged+project:openstack/oslo.messaging+branch:master+topic:bp/transport-options
23:02:25 sean-k-mooney but i dont think anyone has gotten around to using that in nova yet
23:02:43 eandersson Yea - and would probably not be easy to backport.
23:02:45 eandersson Or even possible.
23:03:33 sean-k-mooney well we need to do too things. frist start using it in master and second we need to figure out what the correct action is to take if we get an exception indicating it could not be delivered
23:03:44 eandersson I think I am going to add a log to nova just before it sends that notification to the computes.
23:05:01 sean-k-mooney for example if we could not deliver a message to a compute queue should we disabel that compute node so we dont try to schdule to it agian?
23:05:09 sean-k-mooney shoudl we jsut retry?
23:06:30 sean-k-mooney i actully have no idea what the best path forward would be in that case but we kind of need to figure that out before we can fix the issue
23:06:48 sean-k-mooney we now have the feature that allows use to adress it however
23:11:24 openstack Launchpad bug 1661510 in oslo.messaging "topic_send may loss messages if the queue not exists" [Medium,In progress] - Assigned to Gabriele Santomaggio (gsantomaggio)
23:11:24 sean-k-mooney melwitt: did you ever have time to talk to the oslo folks on how we could use this https://blueprints.launchpad.net/oslo.messaging/+spec/transport-options and specificly the mandaroty flag so we can adress the nova aspect of https://bugs.launchpad.net/oslo.messaging/+bug/1661510
23:12:05 sean-k-mooney i know i havent but it looks like all the patchs are in place on the oslo side
23:28:23 melwitt sean-k-mooney: I haven't talked to them bc like you said, their side is done. it's up to us (me) to figure out how to use it on the nova side and I haven't gotten a chance to dig in to it yet
23:28:54 melwitt I don't yet know where/how to pass it in the nova code
23:29:51 sean-k-mooney ya i just asked on the bug if they could provdie an example
23:30:51 sean-k-mooney i know its something we shoudl be setting when we do a topic send which is i guess a call to a compute node primarly
23:31:11 sean-k-mooney but i dont know what we should do if we are not able to deliver it to the message queue
23:40:40 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-services API policies https://review.opendev.org/648480
23:44:56 gmann johnthetubaguy: this should be ready now. I was checking why project-admin and legacy admin can access the new system reader rule. which was correct because of old defaults(admin_api) are deprecated and still work. This make sure any old token will keep working for changed defaults also.
23:45:29 gmann johnthetubaguy: once you are ok with this then it will complete the first set of change - https://review.opendev.org/648480
23:51:23 openstackgerrit sean mooney proposed openstack/nova master: Block rebuild when NUMA topology changed https://review.opendev.org/687957
23:51:24 openstackgerrit sean mooney proposed openstack/nova master: Disable NUMATopologyFilter on rebuild https://review.opendev.org/689861
23:56:08 efried https://bugs.launchpad.net/neutron/+bug/1855015
23:56:09 openstack Launchpad bug 1855015 in OpenStack Compute (nova) "Intermittent fails since 11/23 with "Multiple possible networks found, use a Network ID to be more specific."" [Undecided,New]
23:56:57 sean-k-mooney has that reappared
23:57:37 sean-k-mooney that is a tempset bug and or we have not confgiured tempet correctly
23:58:32 eandersson If we can’t deliver to the queue I think we should just set to error or move to the next available compute
23:58:52 sean-k-mooney eandersson: set what to error?
23:58:57 eandersson The instance
23:59:12 sean-k-mooney that would not be correct in all cases
23:59:54 sean-k-mooney if the call was to say the instace diagnostics endpoint
#openstack-nova - 2019-12-04
00:00:13 sean-k-mooney we shoudl not put the instance in error state because we were not able to get the diagnositcs right
00:00:17 eandersson sure this would only be for the scheduling event
00:00:29 eandersson since mandatory is per message not per queue
00:01:00 sean-k-mooney its per type of rpc call we do
00:01:15 sean-k-mooney e.g. you set it in your argmuent to send
00:02:59 sean-k-mooney if we pass mandaroy on all topic sends then we woudl ahve to deally with it constently for ever api action that needs to call the compute node
00:03:29 sean-k-mooney if we dont set it gloably for all topic sends we need to consdider how to handel the failure specific for each action
00:06:30 sean-k-mooney eandersson: in the specifc case of server create. if the downcall form the condutor to the compute failed to send you would like us to retry the next compute node correct.
00:07:04 sean-k-mooney if we set the instance to error at that point we would not retry
00:09:27 sean-k-mooney if we did retry i think we would still want to incremetn the failed boot counter for the compute node so the BuildFailureWeigher can avoid that host and log an error at a minium
00:09:59 eandersson Sorry stepping away again. Btw we got a ton of open positions if anyone wants to come and help me work on this stuff. :p
00:10:38 sean-k-mooney no worries im about to call it a night anyway
01:45:01 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Pass the target in os-services APIs policy https://review.opendev.org/676688
01:45:45 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add test coverage of existing admin_actions policies https://review.opendev.org/657698
02:22:14 openstackgerrit Boxiang Zhu proposed openstack/nova master: Make evacuation respects anti-affinity rule https://review.opendev.org/649963
02:27:12 eandersson Restart rmq and rebuilt it. Lets see if the issue goes away.
02:29:04 openstackgerrit Boxiang Zhu proposed openstack/nova master: Fix live migration break group policy simultaneously https://review.opendev.org/651969
03:04:33 openstackgerrit Brin Zhang proposed openstack/nova master: PoC: Support re-configure the delete_on_termination in server https://review.opendev.org/693828
03:24:17 eandersson sean-k-mooney yea I would prefer it to retry and write an error to the logs, but probably cleaner and easier to implement it to stop and put the instance into an ERROR'd state instead.
03:25:03 eandersson I am gonna see if I can set up something like Terraform to better handle instances stuck in BUILDING
03:25:28 eandersson I think out of the box it gets stuck in a weird state, but I am sure there is a way to have Terraform handle issues like this more gracefully.
03:59:10 openstackgerrit jichenjc proposed openstack/nova master: libvirt: avoid cpu check at s390x arch https://review.opendev.org/696228
05:05:47 openstackgerrit jichenjc proposed openstack/nova master: downgrade when host does not support capabilities https://review.opendev.org/697228
05:47:26 openstackgerrit Brin Zhang proposed openstack/nova-specs master: Support re-configure deleted_on_termination in server https://review.opendev.org/580336
06:46:55 openstackgerrit OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.opendev.org/694717
07:37:35 openstackgerrit jichenjc proposed openstack/nova master: libvirt: avoid cpu check at s390x arch https://review.opendev.org/696228
07:56:29 openstackgerrit Luyao Zhong proposed openstack/nova master: support live migration with vpmems https://review.opendev.org/687856

Earlier   Later