Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-03-25
20:50:10 rm_work k
20:50:19 rm_work our symptom was that a late check will catch it, but
20:50:27 rm_work for soft-anti-affinity, it doesn't actually BLOCK it
20:50:35 rm_work which means soft-anti-affinity is pretty useless
20:50:53 rm_work hard-anti-affinity it catches and sends for rescheduling, but our problem was we had reschedules disabled
20:50:58 rm_work we have since fixed that
20:51:16 melwitt yeah. I'm not sure what soft-anti-affinity is for ... give it a try and if not, meh I guess
20:51:25 openstackgerrit Merged openstack/nova master: libvirt: Use virDomainBlockCopy to swap volumes when using -blockdev https://review.opendev.org/696834
20:51:29 rm_work it's supposed to be "best attempt"
20:51:37 melwitt yeah
20:51:40 rm_work so if you are down to only one schedulable HV, it'll still *work* because that's better than nothing
20:51:57 rm_work but in our case, using pack scheduling, it basically never does anything unless it catches up front
20:52:05 rm_work the late-catch will do nothing as it's still "valid"
20:52:11 rm_work which makes it not so useful
20:52:16 melwitt right
20:52:43 melwitt so what did you do? enable some retries?
20:53:17 rm_work switched to hard-aa and set scheduling retries to 3 (which is the original default, i think -- we had set it specifically to 0)
20:58:22 melwitt rm_work: yeah, ok. we do have a gap regarding the default pack scheduling + server group requests as mriedem mentioned earlier, and a way we could deal with that is to do something similar to starlingx where we serialize server group request claims at the scheduler, but we'd need to use a distributed lock since we have multiple scheduler workers. not something we already have in nova so would take more effort to add. would be a spec
20:58:23 melwitt and all
21:20:57 openstackgerrit Merged openstack/nova stable/pike: Improve metadata server performance with large security groups https://review.opendev.org/697523
21:24:29 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-hypervisors policies https://review.opendev.org/715071
21:32:12 mriedem melwitt: tbc, i think the solution for the affinity problem during scheduling likely involves placement as has been discussed for years, not serializing things like starlingx did as a workaround
21:32:28 mriedem but how that would work in placement has always been difficult to model
21:33:08 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Pass the actual target in os-hypervisors policy https://review.opendev.org/715074
21:33:26 mriedem but then placement is your distributed lock :)
21:33:31 melwitt mriedem: yeah, sorry, I was missing that the pack pattern problem with anti-affinity would go away with placement affinity
21:33:48 melwitt this stuff confuses the hell out of me
21:35:31 melwitt so, nix what I said earlier rm_work ^
21:40:39 mriedem "if you have multiple MQs there's no enforcement of affinity for racing requests until affinity support is added to placement" is not quite true, you just need the conductors configured to hit the API DB (compute -> cell conductor -> API DB); devstack doesn't configure the cell conductor with the API DB connection so again, devstack is doing the *ideal* separate setup but not what most (if any) nova deployments are probably d
21:40:39 mriedem ,
21:41:09 mriedem pretty sure most nova deployments just have the api db connection configured everywhere
21:41:22 openstackgerrit Merged openstack/nova stable/stein: nova-live-migration: Ensure subnode is fenced during evacuation testing https://review.opendev.org/713962
21:41:40 mriedem and yeah you have to have reschedules enabled to....reschedule :)
21:42:28 mriedem the only things that do the late affinity check in the computes are server create and evacuate, so you can still violate affinity policy for other moves (unshelve, cold and live migrate)
21:42:37 melwitt yeah ... I was realizing that slowly regarding the difference between database access vs MQ access
21:42:55 mriedem and the only flows that reschedule today from the compute are create and cold migrate/resize
21:43:10 mriedem i think evacuate just fails the operation if you fail the late affinity check
21:44:12 melwitt I can't remember why that "impossible to contact bc MQ" was ever a thing wrt to affinity
21:45:35 melwitt was it before alternate_hosts became a thing maybe?
21:46:35 melwitt sigh ... have to correct my comment yet again
21:54:22 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add test coverage of existing instance usage log policies https://review.opendev.org/715080
21:55:05 mriedem alternate hosts just solved the problem of the cell conductor needing to go back to the scheduler and API DB
21:55:41 mriedem there is actually still an upcall bug there which i don't think got fixed,
21:56:11 mriedem during reschedule the conductor will update the instance.availability_zone for the alternate host and to do that it needs to hit the aggregates table which is in the API DB
21:57:04 melwitt yeah, I mean I had thought in the past it was said that the late affinity check would be impossible due to MQ isolation. but as you explained that's not true. so if that was ever said, I wondered why. it might have been before alternate hosts was added
21:57:29 mriedem err i guess i fixed that https://review.opendev.org/#/q/topic:bug/1781286+(status:open+OR+status:merged)+branch:master
21:58:46 mriedem anytime, it's time to social distance myself into the kitchen, o/
22:01:36 dansmith melwitt: I think what you're thinking of is that we don't currently have a way for the child cell to know about the top-level mq, and I don't think we should
22:02:04 dansmith we do have a separate api_db connection string, so as a hack, child conductors can use that as if they were a top-level conductor to still hit that database,
22:02:18 dansmith which at least reduces the scope of who at the lower level can talk up,
22:02:41 dansmith and since we should be solving this in placement, we can just hang onto the status quo instead of further pollute the model by teaching everyone to call up
22:03:17 dansmith concerned people could give perms to the child conductors to only view the instance_groups and related tables I think, to further limit the scope of what it can see to just what is needed for tht check
22:04:33 melwitt dansmith: thanks ... I think I am thinking of that. but I could have sworn that there was some previously discussed impossibility about it regarding separate MQs, I might have been mixing back before we had alternate hosts, how once you're in the cell you can't call the scheduler again to request a reschedule
22:05:06 dansmith ...because we don't have a way to tell those services about the upper mq like we do for the upper db
22:05:53 melwitt right. yeah, I do understand that. maybe I was thinking back to before we had alternate hosts and became able to reschedule without sharing a MQ
22:06:29 dansmith well, yeah, the alternate hosts thing was the only way we could reschedule without adding a similar upcall
22:06:34 melwitt and had tied that to the late affinity check in my head. I dunno
22:06:36 dansmith for the same reason
22:06:47 dansmith well, it's the same thing of course
22:07:08 dansmith it was just easier to solve that with pre-populating some alternates to chew through, whereas the affinity check is not so easy
22:07:11 melwitt oh ... guh
22:08:22 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Introduce scope_types in os-instance-usage-audit-log https://review.opendev.org/715082
22:20:45 melwitt dansmith: I think what confused me was that prior to alternate hosts, you needed to be able to access the scheduler's MQ to reschedule (right?) ... so an upcall to another MQ. and I didn't process that the late affinity check does not involve needing to upcall to another MQ to work, it only needs to upcall to the API DB
22:21:47 dansmith melwitt: yes, (re)schedule is an rpc call, whereas the affinity check is just a db operation
22:22:18 melwitt right, ok
22:37:41 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-instance-usage-audit-log policies https://review.opendev.org/715085
22:48:19 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Pass the actual target in os-instance-usage-audit-log policy https://review.opendev.org/715089
22:55:34 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-instance-usage-audit-log policies https://review.opendev.org/715085
22:56:52 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Pass the actual target in os-instance-usage-audit-log policy https://review.opendev.org/715089
23:18:31 openstackgerrit melanie witt proposed openstack/nova master: Add info about affinity requests to the troubleshooting doc https://review.opendev.org/715092
23:18:52 openstackgerrit Merged openstack/nova master: libvirt: Use oslo.utils >= 4.1.0 to fetch format-specific image data https://review.opendev.org/710785
23:19:00 openstackgerrit Merged openstack/nova master: remove DISTINCT ON SQL instruction that does nothing on MySQL https://review.opendev.org/705850
23:20:14 openstackgerrit melanie witt proposed openstack/nova master: Add info about affinity requests to the troubleshooting doc https://review.opendev.org/715092
23:49:00 brinzhang_ gmann: I would like we not only catch 4xx error, maybe be we also need get 500, so I would like to keep the exception name to populate the details if it is a non-nova exception
23:51:26 brinzhang_ gmann: for example: as a non-admin, iuf I created server failed because of NoValidHost(500), if I get this message, that I can try again, Otherwise I cannot get nothing useful message
23:51:39 brinzhang_ s/iuf/if
#openstack-nova - 2020-03-26
00:02:54 openstackgerrit melanie witt proposed openstack/nova master: Add info about affinity requests to the troubleshooting doc https://review.opendev.org/715092
00:09:28 brinzhang_ dansmith: I dont know what you would want to say, you mean we should use update method instead of patch method? We were disscussed in irc meetting, at the beginning I wanted to use `PUT /servers/{server_id}/os-volume_attachments/{volume_id}` to implement this feature, but more people dont agree.
00:09:35 brinzhang_ dansmith: Firstly, we need admin_or_owner role to execute this, but the PUT swap volume API default role is administrative.
00:09:44 brinzhang_ dansmith: Secondly, we should make the `volumeId` to optional in the request body, that change is to big, and many limits and changes will be add this PUT swap volume API.
00:09:48 brinzhang_ dansmith: From talked in the irc meetting before freeze blueprint for Ussuri, we reached a letter of agreement and adopted the alternative scheme[1] of seen-k-mooney to achieve this function.
00:09:54 brinzhang_ [1] https://review.opendev.org/#/c/580336/32/specs/ussuri/approved/destroy-instance-with-datavolume.rst@51
00:10:50 brinzhang_ dansmith: I know you didn't participate in the irc discussion at the time, but I don't understand why we can't use the PATCH API?
00:56:37 alex_xu sean-k-mooney: I guess we are missing a patch for evacuate
01:44:04 brinzhang_ alex_xu, sean-k-mooney: I think in the libvirt we are lost the accel_info after evacuate, https://review.opendev.org/#/c/631245/62/nova/virt/libvirt/driver.py@5781
01:45:32 brinzhang_ alex_xu, sean-k-mooney: when we have an evecuate a server, maybe we need try to get the accelerators info for by instance.uuid, rirht?
03:15:29 sean-k-mooney alex_xu: we are not planning to have all ops supported initally so evac can be added later if we block it in the final patch
03:15:44 sean-k-mooney but it would be good to add it sooner rather then later
06:08:56 openstackgerrit Luyao Zhong proposed openstack/nova master: support live migration with vpmems https://review.opendev.org/687856
06:08:57 openstackgerrit Luyao Zhong proposed openstack/nova master: Track orphan instances and error migrations in resource tracker https://review.opendev.org/714653
08:01:28 gibi good morning nova
08:03:48 brinzhang_ gibi: good morning ^^
08:05:18 gibi o/
08:05:29 brinzhang_ gibi: pls add this PATCH API for implement bp/destroy-instance-with-datavolume to today's agenda, I replied dansmith's comments inline, and above in irc
08:06:17 gibi brinzhang_: OK. Will you be able to participate or I could you update me shortly what the disagreemen is about?
08:06:33 gibi s/I//
08:06:39 brinzhang_ gibi: he seems dont want I use PATCH API to do this, but I dont want to have a change.
08:07:27 brinzhang_ gibi: it's too later for mee, I am not able to participate
08:07:59 gibi brinzhang_: OK thanks. I will raise it to find a way forward
08:09:06 brinzhang_ I think this PATCH reched in the irc meeting, without good reason, I hope it will not stop it from moving forward.

Earlier   Later