| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-05-28 | |||
| 13:12:38 | stephenfin | it seems wrong | |
| 13:14:21 | stephenfin | But I can't decide if we should log something special for that (e.g. LOG.info("instance was not scheduled successfully, deleting from database")) or ignore that case entirely (replace the 'else' with 'elif instance.host') | |
| 13:14:45 | stephenfin | lyarwood: it's pretty trivial, but you might have a opinion on that too ^ | |
| 13:18:20 | gibi | stephenfin: re: glance related bug, thanks. | |
| 13:19:03 | gibi | stephenfin: regarding the misleading log message for local delete of unscheduled instance. I'm more for adding an elif and a special debug message. that is a small cost but helps troubleshoting | |
| 13:21:56 | sean-k-mooney | stephenfin: today we dont delete instance form the db if they are not scheulded correctly so im configed by the message | |
| 13:22:11 | sean-k-mooney | we put the instance into the error state | |
| 13:22:28 | stephenfin | "so im configed by the message" huh/ | |
| 13:22:30 | stephenfin | *? | |
| 13:22:38 | sean-k-mooney | *confused | |
| 13:22:49 | sean-k-mooney | the feeling you feel at reading that | |
| 13:22:53 | stephenfin | how? :D | |
| 13:22:54 | stephenfin | haha | |
| 13:23:08 | stephenfin | sorry, let me restate things | |
| 13:23:17 | stephenfin | we put the instance to error state | |
| 13:23:39 | stephenfin | and I see that message when I later attempt to delete the instance | |
| 13:23:49 | stephenfin | host is unset since the instance failed to schedule | |
| 13:24:08 | sean-k-mooney | it lands in cell 0 right? | |
| 13:24:31 | stephenfin | Specifically, I'm seeing it when I purposefully break nova-glance interactions by setting '[glance] api_servers = <non-existent host>' in 'nova-cpu.conf' | |
| 13:24:37 | stephenfin | Yes, with the host field unset | |
| 13:25:23 | sean-k-mooney | right so after its in error when we delete the isntace we see the "instance was not ..." message int the api log? conductor? | |
| 13:25:56 | stephenfin | gmann: I'm seeing this in logs on a standard DevStack deployment | |
| 13:26:03 | stephenfin | UserWarning: Policy os_compute_api:servers:allow_all_filters failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required | |
| 13:26:24 | stephenfin | gmann: Before I go investigating, is this something you've seen already/something expected? | |
| 13:27:30 | gmann | stephenfin: yeah those are expected warning for scope cheks. default value change warning are suppressed | |
| 13:28:21 | gmann | that is because of policy has scope_type but enforce_scope is disabled. | |
| 13:29:27 | stephenfin | gmann: Ack, thanks | |
| 13:30:02 | stephenfin | sean-k-mooney: Sorry, couldn't find the log. I see it in nova-api | |
| 13:30:50 | sean-k-mooney | stephenfin: so i suspect what is happening is sicne you only broke glance it the compute agent config we are trying to actully boot the instance and failing then hitting the retry limit and landing in cell0 with no host set | |
| 13:31:06 | sean-k-mooney | so its not wrong in that we failed to find a host where it can boot | |
| 13:31:17 | sean-k-mooney | stephenfin: and ya i was excpecting it to be in the api log | |
| 13:32:05 | sean-k-mooney | so your wondering if we we should drop the level form warning to info | |
| 13:32:36 | sean-k-mooney | sorry https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2400-L2401 is a different warning | |
| 13:32:56 | stephenfin | sean-k-mooney: not quite. More if we should log something special for the host=None case or not log at all for that case | |
| 13:33:13 | stephenfin | we should keep warning for host!=None | |
| 13:33:19 | sean-k-mooney | ah i was thinking if we log it should be at debug | |
| 13:33:29 | sean-k-mooney | i dont think it should be at warning | |
| 13:33:45 | sean-k-mooney | i dont think info is useful | |
| 13:34:09 | sean-k-mooney | i would be fine with LOG.debug("instance was not scheduled successfully, deleting from database") | |
| 13:34:25 | sean-k-mooney | for the host is None case | |
| 13:35:22 | sean-k-mooney | i think the intent behind the orginal warning stems form the fact the host is down | |
| 13:35:44 | sean-k-mooney | so you might need to check that the local delete actully works proably when you bring it back up | |
| 13:41:15 | stephenfin | gibi: https://bugs.launchpad.net/nova/+bug/1736920/comments/9 | |
| 13:41:15 | openstack | Launchpad bug 1736920 in OpenStack Security Advisory "Glance images are loaded into memory" [Undecided,Incomplete] | |
| 13:44:13 | sean-k-mooney | stephenfin: are you sure that has always been the case | |
| 13:44:32 | sean-k-mooney | e.g. the use of the decorator | |
| 13:45:19 | stephenfin | sean-k-mooney: A quick look at stable/ocata for glanceclient suggests yes | |
| 13:46:04 | sean-k-mooney | so if we are not loadign the entire image into memroy as a list then that means our imaage validation logic might be broken | |
| 13:46:04 | stephenfin | for v2, and for v1 the 'data' function is the only one that returns IterableWithLength too | |
| 13:50:53 | sean-k-mooney | ok so RequestIdProxy is its self an iterable which delegates to the IterableWithLength | |
| 13:53:55 | sean-k-mooney | and its internal iterable is initalised to the resopocne body | |
| 13:54:12 | sean-k-mooney | ok so ya it look like you are right | |
| 13:54:37 | sean-k-mooney | sincei it returns an iterable object not a generator it wont take that brnach. | |
| 13:56:17 | sean-k-mooney | "utils.serious_integrity_iter" they have some interesting names | |
| 13:56:52 | gibi | stephenfin: thanks. your comment make sense | |
| 14:17:20 | openstackgerrit | Liang Fang proposed openstack/nova master: DNM - test Nova UT that depends on os-brick patch https://review.opendev.org/731388 | |
| 14:30:16 | stephenfin | gibi: bauzas: I meant to point this out last week. sean-k-mooney and I put it together. If you get a chance to take a look, let me know what you think/if there's anything to add https://that.guru/blog/the-numa-scheduling-story-in-nova/ | |
| 14:30:58 | stephenfin | We're trying to answers to commonly asked questions like that somewhere | |
| 14:31:00 | bauzas | stephenfin: gibi: I'm a bit on and off today due to a lot of stuff to do (French WC, bug issues, doc issues...) | |
| 14:31:09 | stephenfin | *to dump | |
| 14:31:11 | bauzas | but how can I help ? | |
| 14:31:28 | stephenfin | bauzas: No help needed. More of an FYI about that article | |
| 14:31:36 | bauzas | ahah lol | |
| 14:31:50 | stephenfin | bauzas: You've already seen it internally :) | |
| 14:32:21 | bauzas | seen what ? | |
| 14:32:28 | gibi | stephenfin, sean-k-mooney: nice blog, I've added that to my read queue | |
| 14:32:36 | stephenfin | bauzas: https://that.guru/blog/the-numa-scheduling-story-in-nova/ | |
| 14:32:39 | bauzas | that I helped french associates to get 120€ more ? :) | |
| 14:32:39 | bauzas | that I helped french associates to get 120€ more ? :) | |
| 14:33:27 | bauzas | weirdo, I haven't seen it in our chan :) | |
| 14:39:14 | openstackgerrit | Merged openstack/nova stable/rocky: Update scheduler instance info at confirm resize https://review.opendev.org/730344 | |
| 14:41:35 | openstackgerrit | Dan Smith proposed openstack/nova master: Make libvirt able to trigger a backend image copy when needed https://review.opendev.org/656998 | |
| 14:41:36 | openstackgerrit | Dan Smith proposed openstack/nova master: Plumb image import functionality through our glance module https://review.opendev.org/731550 | |
| 14:44:32 | dansmith | lyarwood: are you around? | |
| 14:45:56 | lyarwood | dansmith: yup | |
| 14:46:26 | dansmith | lyarwood: hey, surely we have some "so you want to configure rbd..." docs in nova, no? I can't seem to find them with half-assed grepping | |
| 14:47:28 | stephenfin | gibi: could I ask you to poke https://review.opendev.org/#/c/712766/ today/this week? | |
| 14:49:50 | lyarwood | dansmith: only https://docs.openstack.org/nova/latest/admin/configuration/hypervisor-kvm.html#configure-compute-backing-storage that I'm aware of | |
| 14:50:08 | lyarwood | dansmith: and that isn't specific to rbd | |
| 14:50:17 | dansmith | lyarwood: yeah that's unsatisfying | |
| 14:50:54 | dansmith | lyarwood: so, I was going to write up some docs about this multistore thing, but it would be weird to start a doc with "so you know everything about normal rbd, let's turn that up to 11" | |
| 14:51:24 | dansmith | lyarwood: are you capable of writing some basicy docs for rbd if I write the second half for multistore, or know anyone that would be good? | |
| 14:51:58 | lyarwood | dansmith: if you can wait a few weeks then yeah I can write that | |
| 14:52:54 | gibi | stephenfin: on it | |
| 14:53:23 | dansmith | lyarwood: okay, that'd be cool.. I'll write the multistore bit and can hang on for you to be around to fill in the first part | |
| 14:53:56 | dansmith | I guess I could try but I think it'd be pretty contrived | |
| 14:54:01 | lyarwood | dansmith: ack cool just remind me once I'm back :) | |
| 15:05:49 | openstackgerrit | Stephen Finucane proposed openstack/nova master: virt: Add 'context', drop 'network_info' parameters for 'unrescue' https://review.opendev.org/730382 | |
| 15:05:50 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add emulated TPM support to Nova https://review.opendev.org/631363 | |
| 15:05:50 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support for resize and cold migration of emulated TPM files https://review.opendev.org/639934 | |
| 15:18:15 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/queens: Reproduce bug 1869050 https://review.opendev.org/731563 | |
| 15:18:15 | openstack | bug 1869050 in OpenStack Compute (nova) queens "migration of anti-affinity server fails due to stale scheduler instance info" [Low,Triaged] https://launchpad.net/bugs/1869050 | |
| 15:19:27 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/queens: Update scheduler instance info at confirm resize https://review.opendev.org/731564 | |
| 15:54:51 | gibi | nova meeting will start in 5 miuntes on #openstack-meeting-3 | |
| 16:07:08 | openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Add libvirt RBD configuration doc https://review.opendev.org/731577 | |
| 16:34:19 | openstackgerrit | Merged openstack/nova master: Handle flavor disk mismatches when resizing https://review.opendev.org/712766 | |
| 16:34:41 | sean-k-mooney | gibi: sorry was distracted did ye wrap up the team meeting early | |
| 16:35:08 | sean-k-mooney | i didnt actlly have anything for the adgenda | |
| 16:35:16 | sean-k-mooney | i just ment to join | |
| 16:36:06 | gibi | sean-k-mooney: yepp it was done early | |