Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-01-28
06:29:53 HagunKim INFO nova.compute.manager [instance: intance id] Terminating instance
06:29:53 HagunKim Here is nova-compute log when a vm is deleted. I removed date, req. id, instance id.
06:29:53 HagunKim Hello, Nova! I have a question. Sometimes my VMs are deleted unexpectedly. What conf or log should I check??
06:29:54 HagunKim INFO nova.compute.manager [instance: intance id] During sync_power_state the instance has a pending task (deleting). Skip.
06:29:54 HagunKim INFO nova.compute.manager [-] [instance: intance id] VM Stopped (Lifecycle Event)
06:29:55 HagunKim INFO nova.virt.libvirt.driver [instance: intance id] Deletion of /var/lib/nova/instances/instance_id_del complete
06:29:55 HagunKim INFO nova.virt.libvirt.driver [instance: intance id] Deleting instance files /var/lib/nova/instances/instance_id_del
06:29:56 HagunKim INFO nova.compute.manager [instance: intance id] Took 23.99 seconds to destroy the instance on the hypervisor.
08:29:07 gibi HagunKim: with the request_id from your log I would grep the nova logs to see what other events lead to the deletion of the instance
08:29:22 bauzas good morning Nova
08:31:23 gibi bauzas: good morning
08:31:44 bauzas gibi: how are you ? I was a bit on and off last week
08:34:04 gibi bauzas: thanks I'm pretty OK, spent half of my last week downstream with helping planning some internal REST APIs. Do you still need to spend a lot of time downstream?
08:34:17 bauzas gibi: this week, nope
08:34:26 bauzas working on specs and the cold migration issue
08:34:32 bauzas (for GPUs)
08:34:51 bauzas if you have things to be reviewed by me, tell me
08:35:30 openstackgerrit Merged openstack/nova-specs master: support live migration with virtual persistent memory https://review.opendev.org/695863
08:36:17 gibi bauzas: I think there is a small test issue in your placement audit https://review.opendev.org/#/c/670112 but besides that it is pretty close. And the placement fixture enahncement top of that needs a squash / respin
08:37:20 gibi bauzas: on my side the qos live migration needs some review. The bootom patch was commented by dansmith so there we need a re-review from dansmith but you can be the second review if you wish https://review.opendev.org/#/c/702261/
08:37:23 bauzas gibi: sure, lemme know which test issue you found :)
08:37:42 bauzas and /me clikcs on the qos live migration series :)
08:38:55 gibi bauzas: there is two test failure in the functional test in the placement audit patch https://834f0f74916383443a4f-1960c95d24ebbbc2bc76dbab48507616.ssl.cf1.rackcdn.com/670112/13/check/nova-tox-functional-py36/61deb16/testr_results.html
08:39:01 gibi bauzas: thanks
08:39:16 gibi bauzas: you can also hit me with patches to review
08:39:49 bauzas gibi: ah shit, I thought it was a functional test race
08:39:55 bauzas since we had issues last week
08:40:23 bauzas gibi: and sure, but my point is that I'd also like to review specs and changes
08:40:42 bauzas gibi: so if you see other changes that you'd get second eyes...
08:41:23 gibi bauzas: sure I will let you know
09:34:40 openstackgerrit Kobi Samoray proposed openstack/nova master: Avoid fetching metadata when no subnets found https://review.opendev.org/679247
09:44:53 openstackgerrit Kobi Samoray proposed openstack/nova master: Avoid fetching metadata when no subnets found https://review.opendev.org/679247
09:48:04 openstackgerrit Kobi Samoray proposed openstack/nova master: Support large network queries towards neutron https://review.opendev.org/704520
11:12:24 openstackgerrit Stephen Finucane proposed openstack/os-resource-classes master: setup.cfg: Configure 'python-requires' https://review.opendev.org/704533
11:12:28 openstackgerrit Stephen Finucane proposed openstack/os-traits master: setup.cfg: Configure 'python-requires' https://review.opendev.org/704534
11:12:29 openstackgerrit Stephen Finucane proposed openstack/os-traits master: setup.cfg: Remove unused 'build_sphinx' configuration https://review.opendev.org/704535
11:17:00 openstackgerrit Vladyslav Drok proposed openstack/nova master: Make RBD imagebackend flatten method idempotent https://review.opendev.org/704330
11:25:41 openstackgerrit Kobi Samoray proposed openstack/nova master: Avoid fetching metadata when no subnets found https://review.opendev.org/679247
11:27:34 openstackgerrit Kobi Samoray proposed openstack/nova master: Avoid fetching metadata when no subnets found https://review.opendev.org/679247
11:27:52 openstackgerrit Kobi Samoray proposed openstack/nova master: Support large network queries towards neutron https://review.opendev.org/704520
12:01:10 HagunKim gibi: Thanks for your reply. But I found nothing with request_id. I will change log level to debug and see it again.
12:05:18 gibi not even in other nova services like nova api?
12:15:42 bauzas HagunKim: just use the os-instance-actions API
12:16:31 bauzas HagunKim: https://docs.openstack.org/api-ref/compute/#servers-actions-servers-os-instance-actions
12:19:07 HagunKim bauzas: Can I use that api for removed instances?
12:19:22 bauzas "Action information of deleted instances can be returned for requests starting with microversion 2.21."
12:19:26 bauzas :)
12:20:07 bauzas then you'll get the req-ids
12:20:24 bauzas so you can either ask again the same API for a specific req-id or just look at logs
12:20:47 HagunKim Oh I will check. Thank you!
12:33:40 stephenfin yo, sean-k-mooney, you probably want to weigh in on https://review.opendev.org/668656 since I _think_ you're of the same mindset as me RE: not forcing people to use placement constructs in nova land
12:34:15 sean-k-mooney i have in the irc discussions and yes
12:35:12 sean-k-mooney i would be oke with the placement syntax but only if we remove support for placement groups syntax form the flavor and stop doing a direct passthough to the api query effectivly
12:36:00 sean-k-mooney e.g. resouce:* would be ok if and only if we have a layer that parses that and then generate a different query based on the other extraspecs but i prefer the hw:* syntax
12:36:08 openstackgerrit Kobi Samoray proposed openstack/nova master: Avoid fetching metadata when no subnets found https://review.opendev.org/679247
12:40:23 stephenfin my arguments slightly different to that. I'm okay with the placement syntax but I don't want it mixed with the legacy stuff. It's too confusing to reason about
12:43:46 alex_xu sean-k-mooney: stephenfin, if we are going to the vision about placement syntax in the future, will it be hard to deprecated those extra specs we added?
12:45:42 alex_xu I'm just thinking, it is vision, agreed by some people, but it is vision, we don't know when it happend, but people afarid we add more stuff which aren't existing in that vision. just thinking of if we have road to transfer to that vision, then we continue add extra specs, then it won't be a problem
12:47:18 sean-k-mooney alex_xu: i guess the only way i can see moving to placement syntax workin is if we remove the group syntax form the flavor
12:47:42 sean-k-mooney so i for won dont ever see use moving to placemnt syntax unless that happens
12:48:26 sean-k-mooney if the resouces:* requests are coupled to the placemetn toplogy directly that is an upgrade issue
12:49:01 sean-k-mooney that said with an indirection laywer between the placment query and the flavor it would be managable
12:49:11 alex_xu I see
12:49:46 sean-k-mooney but to your question i dont think doing hw:* is a blocker to doing resouces:* in the future
12:50:23 alex_xu but at that time, we need to deprecated some extra sepcs like 'hw:cpu_policy', any problem we deprecated one more more extra spec for 'hw:dedicated_cpu_set'(I forget the name)?
12:50:43 sean-k-mooney we can support that for cpu pinning today but i would continue to recommend using hw:cpu_policy=dedicated over resocues:PCPU=x
12:50:44 alex_xu yea, that is what i'm thinking
12:51:34 alex_xu huaqiang: efried dansmith stephenfin ^
12:51:41 sean-k-mooney ya we would have to deprecate both toghter but 1 vs 2 extra specs is not really a big differece
12:51:45 alex_xu it is time pull everybody togetther again :)
12:52:15 sean-k-mooney we would still have to do an online data migration to update the embeded flavor or force the operator to resize every vm in either case
12:53:02 sean-k-mooney that is when we remove support for hw:cpu_policy if we ever do
12:53:12 alex_xu ah, resize, that is terrible
12:53:48 sean-k-mooney ya its the only way to update the embeded flavor today
12:54:23 sean-k-mooney which is why i dont like the idea of useing the placement syntax direcly as it is copled to the placement toplogy and if that chagne we need to either do a resize or an online data migration
12:55:13 alex_xu yea, i see
12:56:01 sean-k-mooney given we want to move pcpus under numa nodes in the next release or two i worry about relying on the placment syntax currently as that will be a toplogy change
12:58:59 alex_xu and we don't know it will happen in next release
13:00:11 sean-k-mooney ya we dont bauzas has reporposed it for this release https://review.opendev.org/#/c/552924/ but we have not really been reviewing it this cycle
13:01:13 sean-k-mooney so i think its unlikly to happen until next cycle at the earliest
13:04:11 openstackgerrit Kobi Samoray proposed openstack/nova master: Support large network queries towards neutron https://review.opendev.org/704520
13:13:05 openstackgerrit Kobi Samoray proposed openstack/nova master: Support large network queries towards neutron https://review.opendev.org/704520
13:13:19 openstackgerrit Kobi Samoray proposed openstack/nova master: Support large network queries towards neutron https://review.opendev.org/704520
13:31:03 openstackgerrit Vladyslav Drok proposed openstack/nova master: Make RBD imagebackend flatten method idempotent https://review.opendev.org/704330
13:32:37 openstackgerrit Ivaylo Mitev proposed openstack/nova master: Faults from cell DB missing in GET /servers/detail https://review.opendev.org/699176
13:36:15 openstackgerrit Ivaylo Mitev proposed openstack/nova master: Faults from cell DB missing in GET /servers/detail https://review.opendev.org/699176
13:40:33 bauzas sean-k-mooney: alex_xu: comments are welcome on the spec
13:40:46 bauzas and I can provide a new revision given gibi's comments
13:41:02 openstackgerrit Ivaylo Mitev proposed openstack/nova master: Faults from cell DB missing in GET /servers/detail https://review.opendev.org/699176
14:08:05 sean-k-mooney bauzas: you spec mainly look at the queries sent to placement not how that is expressed in the flavor but i have conserns with how resouce request are represent in the flavor and i dont really think you should proceed untill hugepage/mempages are supported
14:08:33 sean-k-mooney im somewhat flexible on the last point but im not a fan of moving memory_mb to the numa node as it stands
14:17:22 bauzas sean-k-mooney: yeah I need to think about the group_policy issue
14:23:41 sean-k-mooney its not just group_policy although that is a major issue i dont think the group syntax shoudl be used in the flavor at all
14:24:18 sean-k-mooney using it between nova and placment im more ok with
14:24:19 bauzas sean-k-mooney: in the spec, I say that we should hide the placement queries
14:24:32 bauzas that's the consensus said so far at least
14:26:54 sean-k-mooney yes that is a saving grace of the current proposal in that it is advocatign for doing translation form the existing extra specs
14:27:16 sean-k-mooney that said its in conflict with the mix pinned and unpind cpus in one instnace work
14:55:30 efried stephenfin: whoops, we crossed in the mail, let me go back and respond to your comments before you respin please...
15:09:23 efried stephenfin: done

Earlier   Later